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FOREWORD 


This  report  represents  an  interim  product  of  continuing  studies 
to  develop  a  series  of  concepts  and  procedures  for  system  engineering 
management  of  computer  programs.  Its  purpose  at  this  time  is  to 
present  a  description  of  the  developmental  process  in  a  form  which 
will  provide  a  basis  for  further  expansion  and  detailing  and  for 
interim  contractual  requirements.  The  further  expansion  contemplated 
will  emphasize  additional  description  of  appropriate  technical  docu¬ 
mentation,  taking  into  account  the  precedents  established  by  Air 
Force  Systems  Command  Manual  (AFSCM)  375-5* 

It  is  recognized  that  the  procedures  specified  in  the  current 
AFSCM  375-5  (March  1966)  are  derived  principally  from  experience  in 
weapon  system  and  equipment  engineering.  While  there  is  little 
question  that  the  same  general  precepts  and  objectives  apply  also 
to  information  processing  systems,  the  applicability  of  certain 
detailed  procedures  and  requirements  to  computer  programming,  in 
particular,  has  not  yet  been  determined.  Problems  stem  from  general 
characteristics  of  digital  computer-based  systems,  from  the  variety 
of  ways  in  which  computer  programs  differ  both  functionally  and 
physically  from  items  of  equipment,  and  from  the  fact  that  the  com¬ 
puter  programming  field  does  not  yet  have  the  wealth  of  working 
standards  which  have  evolved  over  decades  in  engineering. 

As  described  herein,  the  technical  milestones  in  computer  program 
analysis,  design,  and  development  are  identified  and  related  to  re¬ 
quirements,  including  documentation,  in  the  areas  of  configuration 
management,  personnel  subsystem,  procedural  data,  system  exercising, 
and  testing,  which  have  been  derived  from  earlier  studies  in  this 
project.  The  project  is  continuing,  directed  towards  improved 
clarification  of  system  engineering  for  computer  programs  along 
the  following  lines: 

1.  This  is  a  preliminary  guide  and,  as  time  and  manpower 
permit,  will  be  revised  to  reflect  improved  integra¬ 
tion  and  results  of  initial  reviews,  and  to  include 
examples  of  system  engineering  documentation. 

2.  Where  indicated,  revised  or  new  AFLC/AFSC  Forms  9 
will  be  recommended  to  cover  system  engineering 
data  requirements  for  computer  programming. 

3.  Recommendations  covering  the  computer  program  area 
will  be  formulated  for  future  inclusion  in  a  revised 
issue  of  AFSCM  375-5  and  associated  exhibits. 
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The  process  and  techniques  described  in  this  report  have  been 
primarily  evolved  as  the  result  of  an  Air  Force  contract  (#F19628-67- 
C-0128,  Management  Procedures  for  Computer  Programming  for  Electronic 
Systems  Division)  with  the  System  Development  Corporation,  Santa 
Monica,  California.  The  principals  at  the  System  Development  Corpora¬ 
tion  were  Lloyd  V.  Searle  and  Robert  L.  Henderson. 

The  Air  Force  technical  management  of  this  project  has  been 
provided  by  the  Technical  Requirements  and  Standards  Office  of  the 
Electronic  Systems  Division,  USAF,  L.  G.  Hanscom  Field,  Bedford, 

Mass.  Guidance,  direction,  and  coordination  were  accomplished 
primarily  by  the  authors  and  Joseph  L.  Pokorney,  now  with  Peat, 

Marwick,  Livingston  and  Co.,  Boston,  Mass.  The  material  in  this 
report  represents  one  phase  of  contractual  support  to  Air  Force 
projects  2801  and  6917,  which  were  funded  and  administered  by  the 
Computer  and  Display  Division  of  the  Deputy  for  Command  Systems, 
Electronic  Systems  Division. 

At  the  System  Development  Corporation,  essential  materials  for 
the  body  of  this  report  were  supplied  by  personnel  of  the  Operational 
Systems  and  Training  Systems  Departments.  Particular  credit  is  due 
to  Eugene  H.  Sydow  and  Stephen  A.  Strohman  for  their  continuing 
support  and  coordination,  as  well  as  for  their  extensive  personal 
contributions  to  the  narratives  of  computer  programming  events 
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ABSTRACT 


This  report  discusses  the  application  of  System  Engineering 
principles  to  the  development  of  computer  programs  and  associated 
elements  of  large  systems.  Using  the  "road  map"  approach  and 
basic  concepts  established  by  Air  Force  manuals  AFSCM  375-h  and 
375-5,  it  describes  step-by-step  events  during  the  conceptual, 
definition,  acquisition,  and  operational  phases  of  a  system  life 
cycle.  Technical  milestones  in  system  and  computer  program  analysis, 
design,  and  development  are  identic ied  and  related  to  associated 
activities  and  requirements  in  the  areas  of  test  and  evaluation, 
configuration  management,  data  management,  and  human  factors. 
Procedures  outlined  in  the  guide  are  based  on  experience  with  a 
number  of  large  electronic  systems  developed  for  the  Air  Force. 
Emphasis  in  the  description  is  placed  on  aspects  of  the  technical 
process  which  have  significant  implications  for  system  program 
planning  and  management.  The  extent  of  contractual  application 
of  the  material  in  this  report  should  be  made  on  a  contract-by¬ 
contract  basis.  The  selective  use  of  the  events,  activities  and 
technical  processes  described  in  this  report  will  assure  a  systemized 
approach  to  the  design,  development  and  acquisition  of  computer- 
based  systems. 
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Chapter  1 


INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  guide  is  to  set  forth  and  describe  key  aspects  of  system 
engineering  management  as  applied  to  information  processing  elements  of 
electronic  systems.  The  scope  and  level  of  treatment  are  generally  comparable 
to  material  contained  in  the  AFSC  Manual  375-5.  However,  the  intent  is  to 
amplify  the  process  for  computer  programs,  in  particular,  as  important  system 
elements  for  which  the  concepts  of  system  engineering  management  have  not 
previously  been  clarified. 

The  material  contained  herein  is  derived  principally  from  experience  with 
L-Systems  as  developed  at  the  Electronic  Systems  Division,  and  from  Air  Force 
concepts,  objectives,  and  requirements  which  are  set  forth  in  the  375-series 
regulations  and  manuals.  While  the  guide  is  designed  to  meet  the  specific 
needs  of  electronic  systems /subsystems  developed  within  that  framework,  it 
intentionally  incorporates  both  system  and  computer  programming  concepts 
which  are  consistent  with  a  wide  variety  of  information  processing  applications. 


B.  SYSTEM  ENGINEERING  MANAGEMENT 


1.  Air  Force  Systems  Management 

Systems  management  is  the  process  of  planning,  organizing,  and  controlling 
the  activities  of  contractors  and  participating  organizations  to  accomplish 
objectives  of  time-phased  system  programs  which  require  the  combined  efforts 
of  diverse  functional  agencies.  As  developed  by  Air  Force  Systems  Command, 
it  is  accomplished  by  a  centralized  System  Program  Office  (SPO),  within  which 
activities  are  structured  into  a  uniform  set  of  management  sub-areas,  each 
identified  as  the  area  of  responsibility  for  an  organizational  unit  of  the 
SPO.  The  management  areas  and  their  general  interrelationships  are  illustrated 
in  Figure  1. 


COMPUTER 
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FIGURE  1.  SYSTEMS  MANAGEMENT. 


2 


As  the  figure  suggests,  system  engineering  management  is  the  foundation  of  the 
systems  management  structure.  It  represents  the  direct  control  of  technical 
efforts  required  to  plan,  analyze,  define,  design,  develop,  test,  and  supply 
the  complex  of  elements  that  constitute  a  system.  To  achieve  an  integrated 
total  program,  it  is  supported  during  Definition  and  Acquisition  Phases  by 
the  efforts  of  configuration  management,  test  and  deployment,  procurement  and 
production,  program  control,  and  logistics.  However,  the  systems  management 
process  as  a  whole  is  characterized  by  its  major  dependence  upon  the  technical 
processes  subsumed  under  system  engineering  as  the  prime  base  for  management 
planning  and  decisions.  Fundamental  to  this  emphasis  is  the  fact  that  timely 
delivery  to  an  operating  agency  of  effective  and  self-sufficient  systems 
embodying  the  latest  national  advances  in  technology  is  a  primary  objective  of 
the  total  process. 

2.  General  Precepts  and  Objectives 

The  term,  system  engineering,  refers  most  directly  to  the  analytical 
and  integrating  activities  which  are  required  to  achieve  a  balanced  design 
for  a  system  as  a  whole,  as  distinct  from  engineering  or  other  scientific 
and  technical  activities  which  are  devoted  to  the  specialized  development 
of  individual  system  elements.  A  modern  complex  system  consists  of  personnel, 
facilities,  computer  programs,  and  data,  in  addition  to  a  wide  variety  of 
equipment.  The  special  objective  of  system  engineering  is  to  assure  that  all 
of  these  parts  are  properly  devised  and  combined,  in  such  ways  that  they  will 
function  efficiently  as  a  unit  in  performing  the  system  mission. 

As  a  formally  recognized  concept,  system  engineering  has  had  most  of  its 
growth  and  application  during  the  past  several  years  in  the  context  of 
weapon  systems.  In  that  framework,  items  of  equipment — e.g. ,  aircraft, 
missiles — are  typically  the  major  elements  to  be  designed  and  developed  during 
the  system  program.  As  such,  they  have  come  to  be  accepted  as  the  "pacing 
items"  for  system  pieinning  and  an  the  reference  base  against  which  the 
compatibility  of  other  elements  can  be  estimated,  evaluated,  and  assured 
during  the  system  engineering  process.  Hence,  many  of  the  particular  objectives 
and  applications  are  most  widely  known  and  understood  at  present  in  relation  to 
their  equipment  engineering  origins. 

It  is  one  purpose  of  this  guide  to  identify  certain  new  interpretations  and 
practices  which  are  appropriate  when  the  pacing  items  in  a  system  or  subsystem 
are  computer  programs  and  personnel,  rather  than  special-purpose  equipment. 
However,  the  modifications  represent  refinements  and  minor  alterations,  on  the 
whole,  rather  than  basic  changes;  in  general,  reference  is  maintained  to  such 
fundamental  concepts  as  the  following: 

a.  Total  System  Design.  The  development  of  an  effective  system  is 
a  complex  process,  involving  such  variables  as  facilities,  equipment,  personnel 
and  training  requirements,  computer  programs,  procedural  data,  installation, 
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testing,  and  special  elements  of  support  equipment,  services,  and  supply.  Inter¬ 
action  among  variables  is  the  rule,  rather  than  the  exception.  The  design  of  a 
display,  for  example,  will  normally  affect  the  requirements  for  knowledges  and 
skills  of  operating  system  personnel,  occasionally  in  a  critical  fashion;  the 
location  of  an  electronics  package  in  an  aircraft  may  affect  not  only  its 
operation  and  aircraft  performance,  but  also  the  requirements  for  ground  main¬ 
tenance  tools,  facilities,  test  equipment,  and  spares.  Thus,  the  approach  to 
developing  a  balanced  system  requires  that  design  decisions  for  components  be 
based  not  only  on  technical  objectives  for  the  component,  but  also  on  a  proper 
consideration  of  its  effects  on  design,  operation,  and  support  of  other  elements 
in  the  system  as  a  whole. 

b.  Uniform  Design  Process .  The  analysis  and  design  of  a  system  and 
its  elements  occurs  at  a  series  of  levels.  Requirements  exist  initially  in  the 
form  of  general  mission  objectives  to  be  met  by  the  proposed  system.  A  first 
step  is  to  identify  the  gross  functions  which  the  system  must  perform  to 
accomplish  those  objectives,  together  with  performance  requirements  (speeds, 
capacities,  tolerances,  limits),  design  constraints,  and  associated  require¬ 
ments  for  system  support  and  development.  Functional  requirements  are  analyzed 
through  progressively  lower  levels  and  allocated  at  each  stage  to  system 
elements  and  subelements  through  decisions  which  take  into  account  feasible 
alternatives  and  their  implications  for  performance  and  design  of  other  elements. 
When  carried  through  systematically  at  all  levels,  this  functional  approach 
serves  to  maintain  visibility  of  each  element's  role  in  the  system  as  a  whole, 
to  clarify  interrelations  among  elements,  and  to  reduce  the  possibility  that 
essential  elements  will  be  omitted. 

As  applied  to  the  life  cycle  of  a  system,  the  system  mission,  functions,  and 
configuration  are  established  in  the  Conceptual  Phase;  end  item  performance 
and  design  requirements  are  analyzed,  defined,  and  selected  in  the  Definition 
Phase;  and  the  system  elements  are  designed,  assembled,  and  tested  during 
Acquisition. 

3.  Precepts  Established  by  AFSCM  375-5* 


The  Air  Force  Systems  Command  Manual  375-5 >  System  Engineering  Management 
Procedures,  defines  detailed  procedures  and  technical  documentation  to  establish 
and  implement  a  uniform  design  process  reflecting  the  above  general  principles 
and  logic.  Since  AFSCM  375-5  provides  the  basic  references  for  material  con¬ 
tained  in  this  guide,  certain  of  its  important  features  are  reviewed  briefly 
below. 


a.  Life-Cycle  Management .  Relationships  among  documentation, 
engineering,  design  reviews,  specifications,  baselines,  and  major  commitment 
points  are  described  on  a  step-by-step  basis  for  the  four  phases  of  a  system 
life-cycle.  Events  and  activities  are  identified  on  a  "road  map"  and  amplified 
by  narratives  for  each  block  shown.  While  key  points  are  related  throughout 
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to  system  program  management  events  established  by  AFSCM  375-4,  emphasis  is 
placed  on  amplifying  the  technical  processes  of  analysis,  definition,  design, 
development,  and  testing. 

b.  Functional  Basis.  A  functional  approach  is  employed  as  the  frame  of 
reference  for  identifying  initial  requirements  at  each  design  level  of  the 
system.  Functions  to  be  analyzed  and  interrelated  for  each  system  are  classified 
under  the  major  categories  of  Operations,  Maintenance,  Test,  and  Activation. 

(1)  Operations  functions  are  the  repetitive  actions  performed  by  a 
system,  following  turnover  to  the  using  agency,  to  accomplish  the  using  agency 
mission  objectives.  These  are  the  primary  system  functions  which  provide  the 
initial  basis  at  each  stage  for  analysis  and  identification  of  requirements  in 
associated  supporting  and  developmental  areas.  Conversely,  they  are  affected, 
validated,  and/or  changed  by  subsequent  effort  to  determine  feasible  and 
effective  solutions  for  Maintenance,  Test,  and  Activation. 

(2)  Maintenance  functions  are  actions  necessary  to  return  a  failed 
system  element  to  readiness  (corrective  maintenance)  or  to  assure  a  continued 
state  of  readiness  (preventive  maintenance).  As  in  the  case  of  Operations 
functions,  these  refer  primarily  to  actions  following  turnover  to  a  using 
agency. 


(3)  Test  functions  are  actions  necessary  to  verify  that  the  system 
and/or  its  elements  meet  established  requirements.  These  are  primarily 
associated  with  the  system  development  and  production,  rather  than  with  user 
operations. 

(4)  Activation  functions  encompass  production,  training,  and  checkout 
actions  necessary  to  procure,  fabricate,  train,  assemble,  handle,  store,  prepare 
for  shipment,  transport,  and  receive  elements  of  the  system.  These  are  normally 
the  initial  and  non-repetitive  actions  during  the  development  program,  preparatory 
to  system,  subsystem,  or  end  item  testing. 

c.  Documentation.  Fundamental  to  the  approach  taken  by  AFSCM  375-5  is 
the  principle  that  systematic  documentation  of  each  step  in  the  technical 
analysis  and  design  process  is  essential  to  both  sound  engineering  practice 
and  sound  management  of  a  complex  system  program.  In  following  this  principle, 
the  manual  defines  a  comprehensive  set  of  specific  technical  documents,  together 
with  standard  format  and  content  requirements,  around  which  to  develop  and 
amplify  details  of  the  system  engineering  process  during  a  system  life-cycle. 

Among  the  total  of  fourteen  documents  defined, the  six  listed  and  described 
briefly  below  are  of  particular  interest  in  relation  to  information  processing 
elements  of  electronic  systems. 

(l)  Functional  Flow  Block  Diagram  (Functional  Diagram).  Functional 
Diagrams  are  drawings  which  structure  and  relate  sequences  of  functional 
operations  which  will  meet  system  requirements.  They  are  drawn  at  a  series 
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of  levels,  beginning  with  top  level  and  progressing  through  first,  second, 
third,  and  lower  levels  as  required  to  identify  and  define  parallel  and 
sequential  interactions  of  necessary  functions.  Their  purpose  is  to  identify 
functional  needs  of  the  system,  independently  of  design  solutions  initially,  in 
the  areas  of  Operations,  Maintenance,  Test,  and  Activation. 

(2)  Requirements  Allocation  Sheet  (RAS).  RASs  are  sheets  used  to 
record  design  requirements  and  constraints  for  each  function  portrayed  on  the 
Functional  Diagrams.  As  key  records  of  Definition  Phase  analyses,  the  RASs 
are  developed  in  a  series  of  steps  and  iterations ,  resulting  in  a  set  of 
listings  which  identify,  for  each  function: 

(a)  the  design  requirements,  in  terms  of  input,  output,  and 
performance  values,  tolerances  and  limits,  structural/design  constraints, 
reliability,  safety,  etc.,  and  functional  or  technical  interfaces; 

(b)  requirements  imposed  on  facilities,  in  terms  of  environ¬ 
ment,  power,  space,  location,  etc.; 

(c)  name,  and  CEI  number  or  other  designation,  of  the  item 
of  equipment  to  vhich  the  function  is  allocated;  and 

(d)  for  functions  which  involve  personnel,  relevant  task, 
skill,  timing,  training,  including  training  equipment,  and  procedural  data 
requirements. 

(3)  Trade  Study  Report.  This  is  a  report  which  documents  alternative 
design  approaches  and/or  design  solutions,  together  with  back-up  rationale 

for  selection  among  alternatives  based  on  technical  capabilities,  cost,  time, 
resource  limitations,  or  other  relevant  considerations  and  constraints.  Trade¬ 
off  studies  to  select  among  significant  alternatives  may  be  accomplished  at 
all  levels  of  system,  subsystem,  and  end  item  analysis  and  design,  backed  up  by 
time-line  analyses  (see  below)  when  appropriate. 

(U)  Time-Line  Sheet.  This  is  a  standard  form  used  to  record  analyses 
of  time-critical  functions,  to  facilitate  visibility  of  sequential  operations 
or  tasks  and  evaluate  design  effectiveness  in  relation  to  constraints  of  timing. 

(5)  Schematic  Block  Diagram.  The  Schematic  Block  Diagram  portrays 
the  solutions  adopted,  or  proposed,  for  allocating  functions  within  the  system, 
subsystems,  and  end  items.  The  diagrams  are  prepared,  identified,  and  controlled 
as  engineering  drawings,  for  inclusion  in  the  system  and  contract  end  item 
specifications.  They  Eire  drawn  at  three  levels: 

(a)  A  first-level  schematic  depicts  functions  apportioned  to 
subsystems,  emd  shows  inter-system  relationships. 
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(b)  A  second- level  schematic  is  a  technical  expansion  of  the 
first-level  diagram,  relating  contract  end  items  within  a  subsystem. 

(c)  A  third-level  diagram  shows  expanded  logic  within  elements 
represented  at  the  second  level.  These  are  prepared  typically  to  reflect  the 
detailed  logic  of  Part  I  CEI  Detail  Specifications. 

(6)  Design  Sheet.  Design  sheets  are  employed  to  integrate  the 
design  requirements  identified  previously  by  functions ,  in  the  RASs,  into 
contract  end  items.  They  define  quantitative  requirements  and  tolerances,  and 
organize  the  criteria  for  subsequent  detail  design  into  the  format  of  sections 
three  and  four  of  the  Part  I  Specifications. 

As  a  set,  the  above  six  documents  constitute  basic  elements  of  the  processes 
of  system-level  and  CEI-level  analysis  and  design  during  Conceptual  and 
Definition  Phases  of  the  system  cycle.  Their  content,  levels,  and  specified 
uses  are  designed  to  insure  technical  integrity  of  the  system  design,  and  to 
furnish  information  by  which  the  adequacy  and  completeness  of  design  can  be 
verified.  When  fully  executed  for  all  items  and  at  all  levels,  they  should 
permit  (a)  tracing  all  derived  requirements  to  their  sources,  and  (b)  correlat¬ 
ing  requirements  among  the  diverse  functional  elements  of  equipment,  facilities, 
procedural  data,  and  personnel. 


7 


c. 


RELEVANT  FACTORS  IN  ELECTRONIC  SYSTEMS 


1.  General 


The  effort  to  define  a  uniform  process  for  the  development  of  systems 
involves  certain  assumptions  about  the  degree  to  which  systems  have  common 
characteristics.  Since  no  two  systems  are  ever  identical,  some  degree  of 
generalization  is  always  necessary.  The  problem  is  to  arrive  at  levels  of 
detail  which  strike  a  proper  balance  between  adequacy  of  guidance,  on  the  one 
hand,  and  flexibility  of  application,  on  the  other.  While  it  has  been  con¬ 
ceded  that  electronic  systems  differ  in  significant  ways  from  weapon  systems, 
as  a  class,  it  is  also  true  that  electronic  systems  themselves  are  by  no  means 
alike.  Currently,  the  Air  Force  recognizes  important  distinctions  among  the 
following  three  subclasses  of  computer-based  (data)  systems. 

a.  Research  and  Development  Supporting  Data  Systems.  These  are 
systems  for  performing  computational  processes  such  as  simulation,  data 
reduction,  test  analyses,  etc.,  which  are  supplied  for  direct  support  of 
approved  research  and  development  activities. 

b.  Management  Supporting  Data  Systems.  These  are  data  processing 
systems  which  support  the  conduct  of  management  or  administrative  functions 
within  functional  organizations  or  commands. 

c.  Operations  Supporting  Data  Systems.  These  include  systems 
which  produce  information  for  decision-making  related  to  direct  command, 
management,  and/or  control  of  forces,  and  those  which  support  weather,  warning, 
intelligence,  and  other  operationally  associated  functions. 

This  guide  is  directly  concerned  with  the  Operations  Supporting,  including 
certain  systems  in  the  Command  and  Control  class ,  since  these  are  currently 
the  types  of  data  systems  to  which  375-series  systems  management  procedures 
normally  apply.  Thus,  the  "model”  is  exemplified  by  such  systems  as  BUIC, 

NORAD  COC,  465L,  Ul2L,  and  others,  as  well  as  by  the  information  processing 
subsystems  of  more  complex  systems — e.g.,  AWAC — in  which  large-scale  informa¬ 
tion  processing  capabilities  are  combined  with  the  development  of  sensors 
and/or  vehicles  in  a  single  system  program.  In  general,  concern  is  with  the 
information  processing  aspects  of  such  systems,  since  these  are  the  prominent 
aspects  to  be  considered  in  distinction  to  aircraft  and  missile  systems. 

In  the  interests  of  uniformity,  it  is  desirable  to  avoid  overemphasizing  the 
uniqueness  of  any  one  system  model,  whether  within  the  context  of  data  systems 
or  weapon  systems.  However,  the  system  engineering  process  is  outlined  herein, 
largely  as  it  has  developed  during  the  past  few  years ,  in  the  light  of  such 
background  considerations  as  the  following: 

a.  Equipment.  The  basic  hardware  in  an  information  system  consists 
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of  a  digital  computer,  or  computers,  and  associated  peripheral  equipment. 

The  technology  for  such  equipment  is  well  established.  While  it  typically 
requires  extensive  contract  effort  during  a  system  program  to  modify,  adapt, 
and/or  configure  to  meet  military  requirements  of  the  given  system,  it  does 
not  normally  require  a  totally  new,  special-purpose  development.  Reliability 
and  maintainability  are  high.  Since  digital  computers  are  basically  general- 
purpose  instruments,  their  detailed  design  is  relatively  independent  of 
specific  system  functions.  Consoles  for  system  operators  tend  to  be  an  excep¬ 
tion,  to  the  extent  that  the  detailed  nature  and  arrangements  of  display/ 
control  elements  must  typically  meet  performance/design  requirements  which 
are  based  upon  detailed  system  operations.  However,  these  are  also  subject 
to  an  increasing  use  of  standard  components. 

b.  Cormunications .  Interfaces  with  other  systems  are  typically 
of  major  importance  in  the  development  of  a  given  new  information  system. 

Matters  to  be  considered  in  this  area  include  message  rates,  volumes,  formats, 
and  types  of  information  to  be  exchanged,  in  relation  to  existing  and  planned 
communications  both  within  and  between  systems. 

c.  Personnel.  Skills  and  knowledges  of  importance  reflect  the  data 
manipulation,  decision-making,  command,  and  resource  management  functions  which 
are  characteristic  of  information  systems.  A  high  proportion  of  system 
operators  are  officer  personnel  of  the  command  organization.  They  work  together 
as  groups,  with  complex  interpersonal  as  well  as  man-machine  interactions.  While 
equipment  maintenance  remains  a  continuing  need,  requirements  are  relatively 
routine  as  compared  with  the  focal  importance  of  this  area  in  most  weapon 
systems.  In  the  support  area,  computer  programming  skills  associated  with 
functions  and  characteristics  of  the  given  system  are  typically  significant. 

d.  Computer  Programs.  System  engineering  considerations  occasioned 
by  the  prominence  of  computer  programs  as  elements  of  information  systems  are 

a  principal  subject  to  be  amplified  subsequently  in  this  guide.  In  general, 
the  technical  development  process  for  computer  programs  involves  a  variety  of 
unique  features  in  comparison  with  other  system  elements ,  having  implications 
at  all  phases  and  levels  of  system  planning,  development,  and  operation. 

e.  System  Exercising.  Requirements  to  establish,  maintain,  and 
evaluate  operational  readiness  are  inherent  in  any  military  system  designed 

to  perform  its  primary  mission  in  a  hostile  environment  which  does  not  actually 
exist  during  much  of  the  system's  operational  life.  These  requirements  are 
most  typical,  in  fact,  of  weapon  systems.  However,  an  operational  exercising 
capability  has  proved  to  be  uniquely  feasible  to  implement  in  certain  electronic 
systems,  where  (l)  the  equipment  can  withstand  long  periods  of  use  without}  serious 
degradation,  (2)  exercises  can  be  conducted  without  compromising  system  readi¬ 
ness  while  they  are  in  progress,  and  (3)  the  nature  of  the  system  mission  and 
operations  permits  realistic  simulation  of  external  inputs,  including  responses 
to  system  outputs .  For  the  most  part ,  these  conditions  have  been  typical  of 
air  defense  systems,  in  which  system  exercising  capabilities  have  had  their 
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widest  application  and  use.  The  feasibility  and  needs  are  not  necessarily 
universal.  However,  in  cases  where  requirements  are  established,  developmental 
implications  must  be  taken  into  account  during  all  phases  of  the  system 
program. 


2.  System  Segments 


As  system  elements,  computer  programs  constitute  a  special  class  of  pro¬ 
ducts  resulting  from  the  system  acquisition  process  which  are  distinguishable 
in  a  variety  of  ways  from  equipment,  facilities,  data,  and  personnel.  Because 
of  significant  similarities  in  requirements  for  their  development,  however, 
they  have  become  established  in  system  programs  as  having  roles  generally 
similar  to  those  of  prime  equipment  and  facilities.  For  example,  they  require 
performance  and  design  specifications  to  be  produced  during  the  system  program 
by  a  contractor;  they  are  subject  to  formal  design  reviews,  inspections,  and 
performance  testing;  and  they  constitute  products  which  require  a  variety  of 
support  items  in  such  forms  as  handbooks,  manuals,  personnel  and  training,  etc.* 

Because  of  their  typical  prominence  in  electronic  systems,  computer  programs 
and  associated  technical  products  are  normally  identified  as  a  subsystem  of 
the  total  system.  With  reference  to  the  concept  of  system  segments  as  set 
forth  in  AFSCM  375-1,  the  work  associated  with  developing  this  subsystem  con¬ 
stitutes  a  distinct  system  segment,  for  which  responsibility  is  assigned  to  a 
single  associate  contractor  or  to  a  single  prime  contractor  in  combination  with 
other  system  segments. 

A  system  segment,  in  essence,  is  an  identified  area  of  contractor  responsibility 
in  a  system  program,  consisting  principally  of  research,  development,  and 
other  activities  associated  with  the  development  and  delivery  of  contract  end 
items  (CEIs)  and  data.  As  discussed  herein,  the  computer  program  segment 
encompasses  the  following  major  activities:  analysis  of  system  information 
processing  functions;  allocation  of  functions  to  operational  personnel  and  the 
computer;  definition  of  performance/design  requirements  for  computer  programs; 
development  of  operating  procedures;  design,  development,  and  testing  of 
computer  programs;  and  development  of  provisions  for  operational  training  and 
other  personnel  subsystem  products. 

Thus,  at  a  minimum,  the  typical  electronic  system  program  will  normally  involve 
the  following  system  segments: 

Computer  Program 

Computer  and  Associated  Equipment 

Communications 

Facilities 


* 

A  detailed  discussion  of  similarities  and  differences  between  computer  pro¬ 
gram  and  equipment  items,  with  respect  to  implications  for  configuration 
management,  is  contained  in  ESD  Exhibit  EST-1,  Section  H. 


While  a  given  system  may  require  other  segments,  e.g.,  sensors,  vehicles,  pro¬ 
pulsion,  guidance,  etc.,  the  principal  areas  of  distinction  with  aircraft  and 
missiles  are  actually  only  the  first  three.  This  guide  attempts  to  clarify 
the  computer  program  system  segment  primarily,  but  with  attention  to  all 
elements  during  the  Conceptual  Phase  and  to  essential  interfaces  with  other 
elements  throughout  the  system  program. 


11 


CHAPTER  II  CONCEPTUAL  TRANSITION 


A.  GENERAL 

The  emphasis  in  this  guide  is  on  those  phases  and  parts  of  phases  of  a  system 
life-cycle  during  which  a  system  program  is  under  formal  control  of  a  System 
Program  Office  (SPO),  under  Air  Force  Systems  Command.  The  narrative  descrip¬ 
tions  and  flow  covered  in  Sections  B  and  C  below  are  confined  to  the  last 
portion  of  the  Conceptual  Phase,  namely,  Conceptual  Transition,  which  represents 
the  initial  stage  at  which  a  SPO  undertakes  to  meet  specific  objectives  which 
are  uniformly  defined  for  all  system  programs. 

Prior  to  Conceptual  Transition,  exploratory  and  advanced  development  studies 
will  have  been  completed,  and  information  will  have  been  acquired  at  levels 
needed  for  issuance  of  the  Requirements  Action  Directive  (RAD)  by  Hq. ,  USAF. 

In  addition  to  Advanced  Development  studies  which  may  have  been  completed 
successfully  to  demonstrate  the  feasibility  of  hardware  or  computer  program 
techniques,  the  prior  studies  should  have  established  the  firm  operating 
needs,  gross  feasibility,  and  a  proposed  system  configuration.  There  are  wide 
variations  in  the  specific  approaches  by  which  this  point  can  be  reached  for 
different  systems.  However,  the  hypothetical  stages  which  are  outlined 
briefly  below  may  illustrate  the  types  and  levels  of  information  which  the 
prior  studies  should  have  yielded,  in  the  case  of  an  information  system. 

1.  Tentative  System  Concept 

At  some  early  stage,  recognized  requirements  for  improvement  in  organiza¬ 
tional  or  command  functioning  have  led  to  an  initial  concept,  or  set  of  alter¬ 
native  concepts,  for  an  improved  operational  capability.  The  nature  of  the 
capability  may  have  been  identified  in  a  preliminary  way,  for  example,  as  a 
computerized  system  containing  a  centralized  base  of  information  required  by  a 
command  organization  or  headquarters  for  controlling  and  managing  forces  and 
resources,  with  real-time  execution  of  identified  control  functions,  or  pro¬ 
visions  for  mathematical  modeling  to  assist  in  planning,  policy-making,  etc. 

At  this  stage,  objectives  are  defined  in  general,  terms  only,  and  functional 
needs  have  not  been  related  to  the  organization’s  complex  mission  in  sufficient 
scope  and  depth  to  provide  a  basis  for  initiating  evaluations  of  the  concept 
or  its  alternatives  in  terms  of  operational,  technical,  and  economic  feasibility. 

A  tentative  system  concept  provides  a  point  of  departure  for  study  and  analysis 
at  succeeding  levels.  As  formulated  initially,  it  may  be  revised,  refined, 
or  possibly  even  abandoned  during  the  course  of  critical  examination  at  later 
points  in  time. 

2.  Analysis  of  Operational  Requirements 

Objectives  at  this  stage  are  to  arrive  at  descriptions  of  the  proposed 
system  mission,  its  operational  environment,  and  interfaces  with  other  systems, 
and  to  formulate  a  doctrine  of  operational  employment.  This  information  is 
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acquired  through  study  of  the  operating  organization  to  identify  and  define 
required  outputs,  inputs,  and  processing  activities  at  a  functional  level — 
i.e.,  without  necessary  regard  to  current  or  projected  design  implementation. 
Implementing  concepts  are  not  excluded  from  consideration;  however,  they 
become  the  principal  subject  of  more  direct  inquiry,  at  progressively  more 
detailed  levels,  during  later  stages.  Information  resulting  from  this  stage 
provides  the  necessary  basis  for  subsequent  study  of  alternative  implementing 
designs.  Emphasis  is  on  identifying  specific  areas  in  which  the  proposed 
system  would  be  expected  to  improve,  or  make  possible,  the  organization's 
performance  of  a  specified  operational  mission. 

There  are  many  methods  and  tools  which  can  be  employed  in  performing  an 
operational  requirements  analysis.  The  magnitude  of  the  task  is  affected, 
initially,  by  the  extent  to  which  requirements  and  deficiencies  in  current 
operations  have  already  been  documented  by  members  of  the  organization.  It  is 
also  importantly  affected  by  size  of  the  organization  and  complexity  of  its 
mission  in  relation  to  functions  contemplated  for  the  proposed  system.  Early 
steps  may  indicate  necessity  to  delimit  the  scope  of  immediate  further  inquiry 
to  some  subset  of  the  mission  elements  which  were  implied  by  the  initial  system 
concept.  In  this  case,  substantial  effort  may  be  involved  at  the  outset  in 
defining,  first,  the  general  parameters  of  an  ultimate  system  to  be  considered 
for  long-range  planning,  and  second,  detailed  parameters  for  the  elements  to 
be  explored  in  depth  and  planned  as  an  initial  capability.  Requirements  for 
compatibility  with  the  ultimate  system  must  then  be  identified  as  continuing 
guidelines  associated  with  the  initial  capability  concept.*  However,  while 
these  and  similar  considerations  will  influence  the  level  and  depth  of 
activities,  the  general  approach  should  encompass  such  points  of  emphasis  as 
the  following: 

(a)  Collect  and  analyze  functional  requirements  reflected  in  approved 
missions/tasks  for  the  organization  and  its  subordinate  units.  Identify 
mission  elements  pertinent  to  the  initial  system  concept;  identify  and  select 

an  approved  subset  of  elements,  as  necessary,  for  detailed  analysis  in  sub¬ 
sequent  steps. 

(b)  Taking  into  account  the  boundaries  implied  by  decisions  made  under 

(a)  above,  collect  and  organize  a  comprehensive  listing  of  the  relevant 
required  outputs,  identifying:  nature  and  levels;  destinations  (internal  and 
external):  requirements  for  accuracy,  sufficiency,  timeliness,  relevancy. 


This  does  not  imply  the  "evolutionary11  development  concept  which  has  been 
propounded  for  some  classes  of  information  systems.  Following  the  issuance 
of  a  RAD  for  the  initial  system,  its  development  will  follow  a  planned  sequence 
of  phases  in  conformance  with  established  procedures  of  system  program  manage¬ 
ment.  While  design  should  permit  flexibility  for  growth  and  incremental 
improvement  during  operations,  it  is  assumed  that  major  increments  of  expansion 
into  the  ultimate  system  might  also  occur  in  a  series  of  programmed  cycles. 
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frequency,  traceability,  etc.  Types  of  outputs  to  be  considered  include 
decisions,  recommendations,  orders,  or  other  information  in  the  form  of 
reports,  forms,  messages,  etc. 

(c)  Collect  and  organize  listings  of  inputs  relevant  to  the  outputs 
identified  above.  Identify  classes,  sources,  accuracy,  currency,  availability, 
and  all  other  characteristics  which  are  related  to  the  dual  objective  of  (l) 
defining  the  inputs  which  the  projected  system  needs,  and  (2)  identifying  the 
degree  to  which  presently  available  inputs  meet,  or  fail  to  meet,  the  needs. 

(d)  Identify  and  define  the  types  and  characteristics  of  information¬ 
handling  and  processing  requirements  associated  with  organizational  outputs. 

These  will  encompass  requirements  for  routine  processes  of  retrieving,  summa¬ 
rizing,  correlating,  validating,  routing,  etc.,  as  well  as  for  complex  modeling, 
real-time  computations,  evaluation,  prediction,  and  other  activities  leading 

to  decisions,  recommendations,  control,  or  other  information  outputs. 

(e)  Document  the  system  operational  requirements.  While  most  data 
collection  occurs  under  steps  outlined  above,  additional  analyses  and  iterations 
occur  in  the  process  of  integrating  the  data  into  a  comprehensive  description 
of  system  requirements.  Quantitative  facts  and  estimates  are  identified  and 
included  where  appropriate.  This  step  involves  further  evaluation,  coordina¬ 
tion,  and  validation  of  requirements  for:  adequacy  in  reflecting  user  needs 
during  the  projected  future  time  period;  attention  to  environmental,  political, 
and  organizational  contingencies;  functional  interfaces  with  existing  and 
planned  other  systems;  major  factors  of  reliability,  flexibility,  security,  and 
state-of-the-art.  Throughout,  attention  is  needed  to  define  specific  technical 
objectives  for  feasibility  studies  to  be  conducted  subsequently,  including 
special  research  in  areas  requiring  long  lead-time  for  technical  solutions. 

3.  Feasibility  Studies 

Feasibility  studies,  in  general,  are  concerned  with  the  evaluation  of  pro¬ 
posed  approaches  and/or  designs  for  implementation.  They  occur  at  different 
levels,  and  with  iterations,  during  all  phases  of  a  system  life-cycle.  At 
this  stage,  they  are  directed  towards  selecting  a  preferred  system  concept — i.e., 
a  gross  system  configuration — through  evaluation  of  alternatives  with  respect 
to  such  major  factors  as  organization,  geography,  and  inter-system  interfaces. 

The  objective  is  to  establish  a  definite  framework  of  major  parameters  and 
boundaries  for  the  selected  system,  as  a  necessary  basis  for  delimiting  the 
potential  scope  of  information  processing  analyses  to  follow.  Together  with 
results  of  the  preceding  requirements  analysis,  results  of  the  feasibility 
studies  should  be  reflected  in  formal  requirements  documentation  (viz.,  the 
RAD)  issued  to  govern  the  conduct  of  subsequent  planning  and  system  analysis. 

For  the  most  part,  the  specific  direction  and  emphasis  of  these  feasibility 
studies  will  be  determined  by  questions  identified  during  the  course  of  pre¬ 
ceding  requirements  analyses.  At  this  level,  a  general  emphasis  can  be 
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expected  on  information  needed  to  guide  or  reflect  policy  decisions  in  such 
areas  as  centralization  vs.  decentralization,  organizational  structure  and 
responsibilities,  deployment  and  operating  dectrines,  etc.  To  varying  degrees, 
studies  or  summaries  will  be  required  to  assess  major  trade-offs  among  relevant 
information  processing  technologies,  including  communications ,  personnel,  and 
support  factors.  Additionally,  state-of-the-art  in  relevant  data  processing 
equipment  and  techniques  should  be  explored  in  sufficient  depth  to  specify  a 
general  design  approach  for  the  system,  considering  the  pertinence  of  such 
techniques  as  time-sharing,  parallel  processing,  remote  query,  predictor 
displays,  rapid  print,  and  others.  Where  special  capabilities  are  being 
considered — e.g.,  mathematical  modeling,  real-time  processing  with  low  frame 
times,  etc. — ,  emphasis  is  needed  to  assess  the  constraints,  feasible  techniques, 
and  associated  concepts  applicable  to  the  performance  of  these  functions,  and 
to  initiate  further  studies  in  depth  where  long  lead-time  for  solutions  can  be 
anticipated. 
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B.  DIAGRAM  OF  THE  CONCEPTUAL  TRANSITION  PROCESS 


The  "Conceptual  Transition  Process"  (Figure  2)  is  a  diagram  depicting 
major  sequential  actions  of  significance  in  system  engineering  which  must  be 
accomplished  prior  to  termination  of  the  Conceptual  Phase.  In  this  phase, 
actions  are  shown  at  two  levels:  (l)  actions  accomplished  by  higher  head¬ 
quarters,  and  (2)  actions  for  which  the  SPO  cadre  is  responsible;  SPO  actions 
may  normally  involve  assistance  by  a  GSE/TDC  and/or  necessary  technical  support 
by  contractors  to  develop  the  required  system  engineering  analyses  and 
documentation . 

The  following  section  consists  of  narrative  descriptions  keyed  to  each  of  the 
numbered  actions  depicted  on  the  diagram. 
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C.  NARRATIVES 

C-l.  Requirements  Action  Directive  (RAD)  Issued 


The  RAD,  prepared  on  AF  Form  71,  is  the  formal  document  issued  by  Hq.  ,  USAF  to 
initiate  Air  Force  actions  necessary  to  begin  a  funded  program  or  project 
leading  to  procurement  of  a  new  system  or  equipment.  In  conformance  with  AFR 
57-1,  17  June  1966,  it  replaces  and  makes  obsolete  the  former  Specific 
Operational  Requirement  (SOR),  Operational  Support  Requirement  {OSR),  and  the 
Advanced  Development  Objective  (ADO).  Where  the  intent  is  to  acquire  a  new 
system,  the  instructions  from  Hq. ,  USAF  to  AFSC  which  accompany  the  RAD  will 
normally  specify  that  the  Conceptual  Phase  will  lead  to  a  Definition  Phase  by 
the  application  of  AFR  375  procedures. 

Thus,  the  RAD  becomes  the  formal  authorization  for  all  subsequent  system 
planning,  design,  and  testing.  From  this  point  on,  the  Conceptual  Phase 
involves  establishing  system  performance  and  design  requirements  and  constraints 
as  necessary  steps  in  constructing  the  Preliminary  Technical  Development  Plan 
(PTDP),  and  to  develop  prerequisites  to  the  Definition  Phase, 

The  RAD  authorizes  the  AFSC  division  to  establish  the  SPO  cadre. 


C-2.  SPO  Cadre  Established 


The  SPO  cadre  is  established  as  the  office  having  full  management  responsibility 
for  the  program  during  transition  from  Conceptual  to  Definition  Phase.  The 
RAD  should  define  future  goals  sufficiently  for  realistic  planning  of  the 
effort  necessary  for  their  achievement.  The  SPO  cadre  is  manned  to  fulfill 
necessary  responsibilities  in  engineering,  procurement,  test,  logistics,  data 
requirements,  personnel  subsystem,  program  control,  and  information  processing. 

Under  leadership  of  the  AFSC  System  Program  Director  (SPD),  the  SPO  cadre 
includes,  in  addition  to  other  AFSC  personnel,  representatives  of  the  using 
command  (e.g.,  SAC,  TAC,  ADC) ,  AFLC,  ATC,  and  other  government  agencies  having 
a  responsible  relationship  to  the  system.  Advanced  planning  support  may  be 
provided  by  key  personnel  from  a  GSE/TDC.  People  comprising  the  SPO  cadre  are 
the  nucleus  from  which  a  formal  SPO  is  established  during  Definition  Phase  A. 


C-3.  Information  Processing  Analysis  Initiated 

Initial  activities  will  normally  include  compiling,  reviewing,  and  assessing 
the  source  documentation  resulting  from  previous  Conceptual  Phase  events  and 
studies.  While  the  preceding  events  may  not  have  followed  a  standard  pattern, 
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they  will  have  included  the  formulation  of  a  system  concept,  the  selection  of 
a  preferred  system  approach  among  competitive  and  alternative  concepts,  and 
the  establishment  of  feasibility  for  a  selected  gross  system  configuration. 
Relevant  exploratory  and  advanced  development  studies  should  have  been  completed, 
together  with  other  necessary  analyses  to  insure  that  the  system  selected  will 
meet  the  operational  performance  capabilities  specified  in  the  RAD  and  will  be 
technically  attainable  within  the  required  time  period,  usually  of  3  to  5  years. 

As  a  critical  first  step,  it  is  typically  necessary  to  verify  that  the  basic 
mission  or  missions  to  be  performed  have  been  identified  and  defined  in 
sufficient  scope  and  depth  to  provide  a  basis  for  the  subsequent  detailed 
analyses  of  information  processing  functions  and  design  requirements.  Additional 
inquiry  and  studies  may  be  necessary  to  insure  that  available  descriptions  of 
mission  requirements  are  complete  and  accurate  with  respect  to  such  areas  as: 
the  military  objectives  and  constraints  associated  with  each  mission  element; 
the  friendly  environment,  including  activation  requirements  and  integration 
with  other  systems;  the  concept  of  operation  associated  with  each  mission/ 
environment;  and  operational  performance  requirements  for  each  mission, 
including  phase-overs  from  one  mission  or  operating  mode  to  another. 

The  Conceptual  Transition  Studies  initiated  at  this  time  are  directed  towards 
assessing  applicable  data  processing  technologies  and  completing  the  identifica¬ 
tion  of  subsystems  and  major  components  by  the  end  of  the  Conceptual  Phase. 

At  that  time,  the  technical  information  relating  to  information  processing 
functions,  equipment,  communications,  facilities,  and  operational  command 
personnel  should  be  available  in  sufficient  depth  to  provide  an  adequate 
technical  base  for  writing  the  PTDP,  as  well  as  for  subsequent  development  of 
the  System  Performance/Design  Requirements  General  Specification  which  will 
be  needed  during  the  early  part  of  Definition  Phase  A. 


C-h.  Develop  System  Functions 

For  the  system  as  a  whole,  the  system  engineering  process  at  this  stage  should 
conform  generally  to  that  outlined  in  AFSCM  375-5  (10  March  1966,  Block  3, 
pp.  28  ff.).  The  process  consists  primarily  in  formulating  a  description  of 
the  system  in  terms  of  functions  which  must  be  performed  in  order  to  meet 
system  requirements,  with  attention  to  such  important  considerations  as: 
insuring  that  continuity  is  maintained  with  established  requirements ,  critical 
decisions  are  documented,  total  system  requirements  are  considered,  means 
are  provided  for  tracing  gross  functions  down  to  detailed  functions  of  elements, 
and  interfaces  among  systems  remain  visible. 

Initially,  system  requirements  are  translated  into  top-level  Functional  Flow 
Block  Diagrams  (Functional  Diagrams),  depicting  the  gross  operations,  mainten¬ 
ance,  test,  and  activation  functions.  Subfunctions  are  then  developed  through 
first-level  Functional  Diagrams  and  to  lower  levels  as  needed  to  define 
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relevant  operations,  maintenance,  and  Category  I  and  Category  II  test  and 
activation  concepts  for  the  given  system.  In  general,  the  attempt  is  made 
to  limit  these  analyses  to  pure  functions,  avoiding  the  use  of  preconceived 
configurations  as  the  basis  for  their  development.  While  exceptions  may 
normally  occur,  objectives  at  this  stage  are  to  assure  completeness  and 
accuracy  of  the  functions,  and  to  provide  maximum  latitude  for  their  subsequent 
allocation  to  optimum  combinations  of  equipment,  facilities,  computer  programs, 
and  personnel. 

With  regard  to  information  processing  functions,  it  may  often  be  assumed  that 
key  aspects  of  the  gross  system  configuration  will  have  been  determined  prior 
to  this  stage,  and  reflected  in  the  RAD.  For  example,  the  types  and  general 
locations  of  sensors,  whether  external  or  self-contained  in  the  system,  and 
other  input  sources  external  to  the  computer  will  be  known;  the  decision  to 
employ  digital  computing  equipment  will  have  been  made;  the  user  organizations ) 
interacting  with  the  computer,  and  whether  at  fixed  or  movable  sites,  will  be 
known;  major  communications  constraints  will  have  been  established;  etc. 

Hence,  emphasis  will  typically  be  placed  on  detailing  the  nature,  frequencies, 
and  volumes  of  input  data;  identifying  required  processing  functions,  and 
defining  the  types,  levels,  and  destinations  of  required  outputs.*  Additionally, 
for  systems  which  involve  a  large  data  base,  an  important  effort  at  this  stage 
will  be  devoted  to  determining  the  data  categories ,  volumes ,  and  storage  require¬ 
ments.  Where  applicable,  this  activity  will  involve  compiling  relevant  informa¬ 
tion  in  such  areas  as  fixed  vs.  variable  parameters,  requirements  for  sites  or 
operating  mode  adaptation,  and  special  requirements  for  data  collection  and 
generation. 

In  addition  to  operations ,  attention  is  required  to  a  variety  of  supporting 
functions  which  must  be  anticipated  and  reflected  in  the  PTDP,  System  Specifica¬ 
tion,  and  System  Test  Plan.  Special  analyses  may  be  required  to  define  functions 
in  the  areas  of  computer  programming  language,  other  utility  functions,  testing, 
simulation,  and  data  reduction.  It  should  be  noted  that  these  will  not  include 
maintenance  functions  for  the  computer  programs  as  such,  although  they  may 
typically  include  maintenance-diagnostic  functions  to  be  performed  by  computer 
programs  for  equipment.  "Maintenance"  does  not  apply  to  computer  programs  in 
the  sense  that  it  applies  to  equipment,  since  computer  programs  do  not  wear 
out  or  otherwise  degrade  as  a  function  of  use,  time,  environment,  etc.  However, 


In  certain  cases,  other  formats  may  be  superior  to  the  "single  thread" 
Functional  Diagrams  described  in  AFSCM  375-5  for  depicting  the  multiple  input- 
output  relations  which  are  typical  of  real-time  processing  systems,  in 
particular.  At  the  same  time,  it  has  been  demonstrated  that  the  AFSCM  375-5 
diagrams  can  be  used,  at  least  to  the  third  level,  as  bases  for  material  to 
be  generated  on  Requirements  Allocation  Sheets.  Further  studies  of  examples 
and  alternatives  for  these  documents  are  currently  in  urofltress. 


19 


since  requirements  for  frequent  changes  during  operational  use  are  typical, 
attention  should  be  given  at  this  time  to  anticipating  the  nature  of  computer 
programming  facilities,  procedures,  and  personnel  which  will  be  required  for 
support  of  the  user  during  the  operational  period. 


C-5.  Determine  Design  Requirements 

Considerations  set  forth  in  AFSCM  375-5  (10  March  1966,  Block  4,  p.  30  ff) 
should  be  taken  into  account  in  developing  the  design  requirements  for  electronic 
systems.  The  process  consists,  in  general,  of  determining  required  character¬ 
istics  of  major  elements  of  facilities,  equipment,  personnel,  procedural  data, 
and  computer  programs  which  will  perform  system  functions  previously  identified. 
Emphasis  is  on  quantified  or  well-qualified  requirements,  supported  by  trade 
study  reports  or  time-line  sheets  where  relevant.  Requirements  are  keyed  to 
functions  identified  on  Functional  Diagrams  at  each  level,  and  are  documented 
on  RASs,  initially,  in  terms  of  such  factors  as:  purpose  of  the  function; 
parameters  of  design  (i.e.,  input  and  output  values  and  allowable  tolerances); 
constraints  such  as  space,  power,  interfaces,  time,  and  environment;  system 
effectiveness,  reliability,  human  performance,  safety,  security,  etc. 

In  the  information  processing  area,  objectives  are  to  (l)  define  performance/ 
design/test  requirements  for  the  information  processing  system/ subsystem  as  a 
whole,  (2)  identify  system  segments,  (3)  allocate  requirements  to  system  segments 
and  to  major  system  elements,  including  personnel  as  well  as  major  elements  of 
computer  programs  and  computing  equipment,  and  (U)  define  all  relevant  inter¬ 
faces  between  segments  and  with  other  systems. 

Together  with  documentation  resulting  from  activities  outlined  in  the  two  pre¬ 
ceding  Blocks,  C-3  and  C-4,  technical  information  compiled  prior  to  the  end 
of  Conceptual  Transition  should  include  coverage  of  the  areas  listed  below. 

(1)  A  verified  and/or  amplified  description  of  system  mission  require¬ 
ments,  covering  mission  objectives  and  constraints,  environment,  integration 
with  other  systems,  operational  and  maintenance  concepts,  phase-overs, 
operating  modes,  and  other  relevant  mission  conditions. 

(2)  Definition  of  system  information  processing  functions,  covering 
external  inputs  and  sources,  processing  operations,  and  outputs  and  destinations, 
together  with  associated  gross  performance/design  requirements  in  terms  of 
volumes,  rates,  timing,  accuracies,  and  special  conditions. 

(3)  A  description  of  the  system  data  base,  including  nature,  volumes, 
storage,  adaptation,  and  special  requirements  for  collection  and/or  generation. 

(1+)  A  description  of  selected  techniques  of  computer  programming  to  be 
adopted  for  the  system — e.g.,  heuristic  programming,  adaptive  control, 
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higher-order  language,  graphic  analysis,  time  sharing,  etc.  A  gross  descrip¬ 
tion  of  the  size  and  operating  characteristics  of  the  computer  program  sub¬ 
system  should  be  included.  This  description  is  based  upon  the  selected 
computer  programming  techniques ,  together  with  system  functional  and  data  base 
analyses,  and  should  include  identified  requirements  in  the  areas  of  mainten¬ 
ance-diagnostics,  utility,  and  support  (e.g.,  test,  simulation,  data  reduction). 

(5)  A  description  of  the  command  organization,  identifying:  levels  of 
command,  responsibilities,  and  geographic  locations;  operator  positions  and 
responsibilities  for  man-machine  functions;  modes  of  display  data  presentation 
(console  digital  or  situational,  closed-circuit  TV,  wall  display,  hard  copy, 
etc.);  required  operator  inputs  to  the  computer;  and  required  communi cations. 

The  description  should  include  a  preliminary  estimate  of  personnel  requirements, 
including  computer  programming  and  simulation/exercising,  as  well  as  operational 
personnel. 

(6)  A  description  of  requirements  for  system  exercising.  This  description 
is  derived  principally  from  analyses  of  user  command  requirements  for 
system/subsystem  evaluation  and  operational  readiness  training  which  can  be  met  by 
periodically  exercising  the  operational  system.  Based  upon  identified  functional 
requirements  (exercising  configurations,  conditions,  missions,  frequencies, 
functional  simulation,  recording,  and  analysis)  and  studies  of  feasibility,  the 
description  should  include  a  preliminary  identification  of  the  major  elements 
required  to  implement  the  exercising  capability.  These  elements  should  include 
additional  demands  on  the  operational  system  (e.g.,  computer  storage  space, 

tape  units,  communications),  as  well  as  special  equipments,  support  computer 
programs,  and  personnel  required  to  perform  the  functions  of  planning,  prepar¬ 
ing,  and  conducting  exercises. 

(7)  A  description  of  interfaces  with  other  systems.  This  description 
should  define  the  requirements  for  inter-system  data  transfer  for  each  mission, 
including  the  resulting  impact  on  other  systems  from  operation  of  the  proposed 
system.  Emphasis  should  be  placed  on  identifying  message  types,  frequencies, 
and  volumes  in  relation  to  required  automatic  and  voice  communications  links. 

(8)  Computer  and  associated  equipment  requirements.  While  this  descrip¬ 
tion  should  be  at  a  level  which  permits  maximum  latitude  for  subsequent  selec¬ 
tion  among  design  alternatives,  limiting  characteristics  should  be  defined  for 
the  following  components  and  parameters :  general  logical  and  physical  equipment 
configuration  and  geographic  locations;  data  processing  speed  and  requirements 
for  special  instructions  or  characteristics ;  estimated  storage  requirements , 

in  terms  of  bits,  sample  word  structures,  type  of  access,  and  access  time; 
input/output  interfaces,  including  rates  and  special  interface  functions; 
requirements  for  peripheral  units  such  as  magnetic  tape  units,  card  machines, 
etc.,  in  terms  of  numbers,  capacities,  and  speeds;  operator  console  numbers, 
types,  input  control  requirements,  and  display  types  and  capacities;  require¬ 
ments  for  special  synthetic  signal  or  message  generating  equipment;  numbers, 
capacities,  and  types  of  special  consoles  for  simulation;  special  displays 
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(e.g.,  large  wall)  or  hard  copy  printouts;  and  growth  potential,  specified 
in  terms  of  the  preceding  items,  reflecting  anticipated  future  expansion  or 
modification  of  system  functions. 


C-6.  Provide  Inputs  to  Preliminary  Technical  Development  Plan  (PTDP) 

The  PTDP  is  prepared  in  the  format  prescribed  for  the  System  Package  Program 
(SPP)  in  AFR  375-**,  which  specifies  sixteen  sections  under  which  system  program 
information  is  to  be  provided  for  review  and  approval  by  higher  headquarters. 

At  this  time,  a  seventeenth  section,  entitled  "  Definition  Phase  Plan",  is  also 
required  for  specific  planning  with  reference  to  the  ensuing  Definition  Phase 
activities. 

Technical  inputs  supply  key  portions  to  be  included  in  many  sections  of  the 
PTDP,  which  becomes  the  governing  authority  for  the  system  program  during  the 
Definition  Phase  and  the  basis  for  subsequent  updating  in  the  form  of  the 
Proposed  System  Package  Plan  (PSPP)  and  the  SPP.  In  general,  inputs  will 
include:  the  diagrams  of  functions  which  will  meet  requirements  specified 
in  the  RAD,  together  with  their  engineering  descriptions;  design  requirements 
derived  from  the  functions;  the  system  configuration  defined  by  Conceptual 
Phase  Studies;  significant  trade-offs  performed  and  identification  of  significant 
trade-offs  remaining  to  be  performed,  with  emphasis  on  areas  of  high  technical 
risk;  and  activation  and  test  concepts  which  will  provide  a  basis  for  subsequent 
writing  of  the  System  Test  Plan  and  Section  k  of  the  System  Specification. 

For  electronic  systems,  inputs  should  establish  that  hardware  parameters  and 
constraints,  and  other  types  and  levels  of  technical  information  outlined  in 
the  preceding  Block  C-5»  have  been  defined  sufficiently  to  permit  initiating 
the  development  of  Part  I  Specifications  for  computer  programs  at  an  early 
point  in  the  Definition  Phase. 

In  addition  to  inputs  based  upon  system  engineering  accomplished  during  the 
Conceptual  Phase,  Section  17.8  of  the  Definition  Phase  Plan,  entitled  "System 
Engineering  Implementation  Plan  (SEIP)",  calls  for  information  covering  the 
implementation  of  requirements  cited  in  AFSCM  375-5.  The  section  covers  plans 
for  application  of  requirements  of  Exhibit  I,  system  engineering  documentation, 
design  reviews,  intersystem  and  intrasystem  trade  studies,  and  system  engineer¬ 
ing  responsibilities  of  in-house  agencies  and  contractors.  Normally,  these 
requirements  are  to  be  applied  to  a  system  and  its  parts  in  a  selective  manner, 
defining  specific  areas  in  which  AFSCM  375-5  procedures  will  be  applied  in  full 
or  in  part  to  each  area.  In  general,  the  emphasis  on  completeness  of  analytical 
documentation  will  be  greater  for  those  areas  and  elements  which  are  new,  and/ 
or  critical,  than  upon  those  which  represent  established  previous  practice 
and  experience.  Also,  where  the  needs  exist,  consideration  may  be  given  to 
alternative  documentation  proposed  by  the  contractor  which  may  not  conform 
to  the  format  and  content  of  AFSCM  375-5  documents  but  which  can  be  shown 
to  accomplish  comparable  end  results. 
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CHAPTER  III 


DEFINITION  PHASE 


A.  GENERAL 

1.  Activities  during  the  Definition  Phase  are  structured  into  three 
sequential  subphases  which  are  designated,  for  simplicity,  as  Phase  A,  Phase 

B,  and  Phase  C. 

(a)  Phase  A  begins  with  the  issuance  of  an  approved  PTDP  and 
associated  system  program  documents  by  Hq. ,  USAF.  It  consists,  in  turn,  of 
three  subsubphases  which  may  extend,  collectively,  for  a  maximum  duration  of 
8  months ,  as  follows : 

(1)  In-house  technical  effort  to  prepare  Work  Statements, 

System  Specification,  plans,  schedules,  and  the  RFP. 

(2)  A  period  of  contractor  proposal  preparation  in  response 

to  the  RFP. 

(3)  In-house  proposal  evaluation/source  selection,  and 
Phase  B  contract  negotiations. 

(b)  Phase  B  represents  the  period  of  contractor  technical  definition 
and  preparation  of  firm  proposals  for  system  acquisition,  designated  by  higher 
headquarters  for  a  maximum  duration  of  6-8  months. 

(c)  Phase  C  consists  of  proposal,  evaluation,  source  selection,  PSPP 
preparation,  evaluation  and  approval  of  the  program  by  higher  headquarters, 
and  negotiation  of  Acquisition  Phase  contracts,  over  a  4-5  month  maximum 
period. 

2.  In  this  guide,  events  depicted  on  the  diagram  (Figure  3)  and  discussed 
in  the  narratives  below  for  Phases  A  and  C  represent  system-level  events  which 
are  performed  by  the  SPO.  In  Phase  B,  although  key  interfaces  with  other 
segments  are  identified,  description  of  the  technical  process  is  limited  to 
activities  and  products  within  the  computer  program  system  segment.  It  is 

to  be  presumed  that  parallel  Phase  B  activities  for  other  segments  of  the 
system  as  a  whole  could  be  shown  at  a  comparable  level,  each  concerned  with 
appropriately  different  specific  tasks  in  the  same  general  areas  of  analysis, 
design,  test  planning,  and  human  factors. 

i 

3.  General  objectives  of  Phase  B  for  the  system,  as  set  forth  in 
AFSCM  375-4,  are  reflected  in  the  nature  and  levels  of  required  Phase  B 
products  which  are  defined  for  the  segment.  The  contractor  final  report, 
as  a  whole,  should  conform  to  the  four -part  outline  defined  in  AFSCM  375-4, 
in  which  Part  II,  "Technical  Report",  contains  the  major  products  of  Phase  B 
system  engineering  effort  (see  Block  D-33).  Included  among  these,  as  key 
products  of  information  processing  analyses  accomplished  up  to  this  time,  are 
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the  Part  I  Detail  Specifications  for  computer  program  CEIs, 


4.  Terms  which  will  be  found  in  the  narratives  to  distinguish  among 
subclasses  of  computer  programs  are  defined  as  follows: 

(a)  Operational.  These  are  computer  programs  designed  to  perform 
functions  which  are  required  to  carry  out  the  primary  operational  mission(s) 
of  the  system. 

(b)  Support.  Support  computer  programs  perform  such  functions  as 
testing,  recording,  data  reduction,  and  simulated  data  generation.  The 
significance  of  distinguishing  these  particular  programs  as  a  subclass  of 
"supporting"  computer  programs  in  general  lies  in  the  fact  that,  while  they 
are  not  essential  to  direct  performance  of  the  operational  mission,  their 
design  is  closely  dependent  upon  design  of  the  operational  computer  programs. 

(c)  Utility.  These  are  computer  programs  used  to  assist  in  the 
operation  of  the  computer,  for  such  purposes  as  assembling,  modifying,  running, 
or  manipulating  other  computer  programs.  As  a  general  rule,  their  design  is 
relatively  independent  of  the  operational  computer  programs  which  are  designed 
to  perform  specific  mission  applications  of  the  computer. 

(d)  Maintenance-Diagnostic.  These  axe  computer  programs  which  also 
assist  in  operation  of  the  computer,  but  for  the  specific  purposes  of  maintain¬ 
ing  the  computer  in  an  operating  state  and  aiding  diagnosis  of  malfunctions. 
Their  design  is  based  on  the  detailed  design  of  the  computer  circuitry. 

However,  they  may  also  involve  operating  and  design  interfaces  with  the 
operational  computer  programs  in  some  real-time  systems. 

The  above  terms  have  been  used  in  a  limited  class  of  systems,  and  are  currently 
referred  to  in  a  number  of  Air  Force  documents.  However,  like  many  other  terms 
and  concepts  to  be  found  in  the  "software"  field,  they  are  by  no  means  standard. 
Disagreements  exist  with  the  definitions  given  above;  and  quite  different 
classifications  are  made  in  some  systems,  e.g.,  non- functional,  applications, 
etc.  The  terms  are  used  herein  largely  for  convenience  in  distinguishing 
certain  phasing  differences  and  design  dependencies  which  tend  to  be  typical. 

It  is  not  intended  to  suggest  that  they  are  universally  applicable. 

With  regard  to  technical  implications,  the  labels  refer  more  directly  to 
classes  of  functional  elements  than  to  computer  program  contract  end  items 
(CPCEIs).  For  example,  an  "operational"  CPCEI  often  includes  certain  simula¬ 
tion  or  data  reduction  functions.  In  fact,  the  complete  set  of  all  operational 
and  supporting  functions  may  be  combined  into  a  single  CPCEI  for  the  system, 
in  the  event  that  all  elements  are  being  developed  by  a  single  contractor  and 
are  intended  for  common  use  and  management  during  the  Operational  Phase  (see 
also  comments  in  Block  D-l  footnote  and  in  Block  D-26). 
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It  may  also  be  noted  that  the  distinctions  correspond  only  partially  with  the 
AFSCM  375-5  classification  of  functions  into  operations,  maintenance,  test, 
and  activation.  Very  briefly:  operations  and  operational  are  essentially 
equivalent;  test  functions  exist  in  both  utility  and  support;  activation  is  not 
of  particular  significance  to  computer  programming;  and  maintenance  has  a  variety 
of  complex  interpretations,  with  some  (different)  applicability  to  both  utility 
and  maintenance-diagnostic. 

For  simplicity  of  discussion,  and  because  their  definitions  are  not  rigorous 
either  in  a  technical  or  in  a  management  sense,  the  variety  of  functions 
contained  in  utility,  support,  and  maintenance-diagnostic  areas  are  grouped 
together  in  the  Definition  Phase  diagram  (Figure  3)  and  referred  to  elsewhere, 
collectively,  as  "supporting"  functions  or  CPCEIs. 
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B.  DIAGRAM  OF  THE  DEFINITION  PROCESS 


1.  The  "Definition  Process",  Figure  3,  depicts  major  actions  of 
significance  which  should  typically  be  accomplished  during  Phases  A,  B,  and  C. 
Phases  A  and  C  consist  principally  of  sequential  system-level  activities 
accomplished  by  the  SPO,  while  Phase  B  events  represent  activities  of  a  con¬ 
tractor  responsible  for  the  computer  program  system  segment. 

2.  The  different  horizontal  levels  at  which  events  are  shown  on  the 
diagram  distinguish  in  a  gross  way  among  subactivities  associated  with  (a) 
development  of  Part  I  Specifications  for  operational  CPCEIs,  (b)  human 
engineering  and  personnel  requirements  development,  (c)  system  exercising 
capability  development,  (d)  test  analysis  and  planning,  and  (e)  development 
of  Part  I  Specifications  for  CPCEIs  in  supporting  areas.  The  location  of  an 
event  at  one  point  in  the  sequence  is  not  meant  to  imply  that  the  event  begins 
and  ends  at  that  discrete  time,  but  to  assist  in  illustrating  the  major 
dependencies  which  are  depicted  by  connecting  lines.  It  is  to  be  assumed  that 
most  of  the  activities  shown  are  in  fact  continuous,  and  that  they  also  interact 
on  a  continuing  basis,  throughout  the  duration  of  Phase  B. 
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C.  NARRATIVES 

PHASE  A  -  PREPARE  FOR  CONTRACTOR  DEFINITION 
D-l.  Expand  System  Analysis  and  Definition 

Technical  studies  to  be  accomplished  during  the  early  part  of  Phase  A  consist 
primarily  of  reviewing,  verifying,  expanding,  and  altering  the  technical  con¬ 
cepts  and  data  resulting  from  Conceptual  Phase  studies,  taking  into  account 
firm  decisions  and  changes  reflected  in  the  approved  PTDP.  Necessary 
expansion  or  alteration  of  functions,  review  and  verification  of  design  require¬ 
ments,  and  performance  of  trade-offs  for  solutions  to  unresolved  alternatives 
are  accomplished  concurrently.  At  this  point,  immediate  objectives  are  to 
establish  the  spectrum  of  operations,  maintenance,  test,  and  activation  functions, 
together  with  associated  design  requirements  and  constraints,  at  levels  required 
for  the  System  Specification  and  other  documents  to  be  issued  with  the  RFP. 

In  the  information  processing  area,  attention  is  needed  to  analyses  which 
will  assure  that  minimum  essential  interfaces  between  computer  programs, 
computing  equipment,  communications  links,  facilities,  and  personnel  are 
adequately  defined.  Parameters  to  be  firmly  established  will  include,  for 
example:  types  and  capacities  of  computer  storage;  timing,  types  and 
capacities  of  display  and  input/output  equipment;  types  and  functional 
characteristics  of  special  simulation  equipment;  expected  data  links,  types, 
formats,  and  rates;  gross  sizes  and  characteristics  of  operational,  utility, 
support,  and  maintenance-diagnostic  computer  programs,  including  design 
requirements  relating  to  timing,  data  sources,  and  quantities;  use  of  higher- 
order  language,  modes  of  display  data  presentation;  and  simulation.  Firm 
information  in  these  categories  is  essential  to  the  subsequent  initiation  and 
successful  completion  of  Part  I  Computer  Program  CEI  Specifications  during 
Phase  B. 

This  effort  leads  to  the  preparation  of  the  initial  specification  tree  for 
the  system.  For  computer  programs,  the  specification  "tree"  is  essentially 
a  list  of  the  CPCEIs;  it  will  not  normally  reflect  levels  of  assembly,  except 
to  the  extent  that  it  may  in  some  instances  contain  one  indenture  to  depict 
government-furnished  items  tentatively  designated  for  assembly  into  major 
CPCEIs.  For  all  CEIs,  the  list  will  be  refined  and  updated  as  a  result  of 
subsequent  contractor  proposals  and  Phase  B  studies.* 


*It  should  be  noted  that  the  selection  of  CEIs  is  based  only  partially  on  tech¬ 
nical  considerations  (cf.  AFSCM  375-1,  Exhibit  XI).  As  a  contracting  concept 
in  essence,  the  CEI  represents  a  level  of  assembly  for  management  control,  to  be 
defined  individually  in  each  case.  Also,  technical  content  is  often  secondary 
to  anticipated  configuration  management  following  acquisition;  for  example,  even 
though  largely  identical  in  initial  content,  two  computer  programs  destined  for 
separate  operational  users  should  be  identified  as  separate  CPCEIs  if  the 
users  are  expected  to  manage  subsequent  changes  independently. 
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D-2.  Prepare  Initial  System  Performance/Design  Requirements  General 

Specification 

The  System  Specification  is  prepared  at  this  time  on  the  basis  of  the  technical 
information  resulting  from  preceding  system  engineering  studies,  in  conformance 
with  content  requirements  set  forth  in  Exhibit  I  of  AFSCM  375-1.  It  includes 
system  performance  and  design  requirements,  system  segment  allocations  and 
interfaces,  identification  of  major  CEIs,  and  requirements  for  system  testing. 
From  the  time  of  its  subsequent  issuance  as  part  of  the  RFP,  it  becomes  a 
controlled  document  which  can  be  changed  only  through  formal  action  of  the 
Configuration  Control  Board  (CCB)  of  the  SPO,  in  accordance  with  the  procedures 
specified  in  Exhibits  VII  and  VIII  of  AFSCM  375-1.  However,  its  use  and  the 
adequacy  and  accuracy  of  its  technical  content  remain  the  responsibility  of  the 
SPO  deputy  director  for  engineering,  with  support  by  a  GSE/TDC.  Hence,  it 
serves  throughout  the  remainder  of  the  system  program  as  a  key  document  for 
both  technical  and  contractual  management. 

Parts  of  the  System  Specification  as  prepared  by  the  SPO  during  Phase  A  are 
incomplete  initially,  and  remain  to  be  expanded  by  the  results  of  contractor 
studies  during  Phase  B.  Data  to  be  eventually  incorporated  in  separate  sub¬ 
sections  under  !TSystem  Allocations"  (paragraph  3.3),  are  developed  separately 
for  each  system  segment  by  the  responsible  Phase  B  contractors  and  submitted 
with  contractor  Phase  B  Final  Reports  (see  Block  D-33). 

In  information  systems,  computer  programming  is  normally  identified  as  a 
separate  system  segment,  for  assignment  either  to  a  prime  contractor  in 
combination  with  other  system  segments  or  independently  to  an  associate 
contractor.  Associated  technical  responsibilities  for  the  system  segment 
will  include  the  varieties  of  R&D  activities  and  contractor  data  products 
which  are  outlined  subsequently  in  this  guide. 

Where  requirements  for  a  system  exercising  capability  have  been  established, 
it  should  be  recognized  that  developmental  responsibilities  will  generally 
parallel  those  for  the  system  itself.  Hence,  performance/design  requirements 
for  the  capability  as  a  whole  should  be  specified  as  a  separate  section  of 
subparagraphs  under  paragraph  3.1  of  the  System  Specification.  Developmental 
responsibilities  for  the  necessary  additional  and  special  equipment,  computer 
programs,  communications,  facilities,  and  associated  data  are  then  allocated 
among  system  segments  along  lines  corresponding  to  allocations  for  the  system. 

Applicable  Data  Item 

C- 1-35. 1-1  System  Performance /Design  Requirements  General  Specification 
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D-3.  Prepare  Initial  System  Test  and  Activation  Plans 

The  System  Test  and  Activation  plans  prepared  at  this  time  by  the  SPO  contain 
the  planning  information  required  to  guide  the  contractors '  activities  related 
to  test  and  activation  during  the  Definition  Phase. 

System  Test  Plan 

The  System  Test  Plan  is  based  upon  the  test  concepts  and  requirements  con¬ 
tained  in  the  PTDP,  the  System  Specification, and  related  system  engineering 
documentation.  It  covers  all  aspects  of  the  system,  including  hardware, 
facilities,  computer  programs,  personnel,  and  procedural  data  with  respect  to 
such  considerations  as  the  following: 

(1)  Organizational  responsibilities  for  testing. 

(2)  Basic  system  test  concepts  and  objectives  for  Category  I,  Category 
II,  Implementation, and  Acceptance  tests. 

(3)  Overall  system  test  operations,  including  test  control  requirements 
and  test  support  requirements. 

(k)  Test  evaluation  requirements. 

(5)  Test  reporting  requirements. 

(6)  Overall  test  schedules. 

A  primary  function  of  the  System  Test  Plan  is  to  guide  the  contractors'  plan¬ 
ning,  analysis,  and  system  engineering  activities  related  to  the  test  areas. 
During  Definition  Phase  B  the  contractors  expand  basic  planning  and  guidance 
information  contained  in  the  System  Test  Plan  and,  in  effect,  replace  it  with 
the  following  documents: 

(l)  Category  I  Test  Plan. 

(2)  Inputs  to  the  Category  II  Test  Plan. 

(3)  Inputs  to  the  Implementation  Test  Plan  (multi-site  systems). 

System  Activation  Plan 

The  System  Activation  Plan,  like  the  System  Test  Plan,  is  based  upon  the 
contents  of  the  PTDP,  the  System  Specification, and  related  system  engineering 
documentation.  It  identifies  basic  objectives,  responsibilities,  operations, 
controls,  logistic  support, and  schedules  for  site  selection,  facilities  con¬ 
struction,  equipment  and  computer  program  installation  ai  d  checkout,  and  system 
demonstration  and  turnover. 
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The  contractors'  planning,  analysis,  and  system  engineering  activities  during 
the  Definition  Phase  are  guided  by  the  contents  of  the  System  Activation  Plan. 
During  Phase  B  the  contractors  expand  upon  the  planning  and  guidance  informa¬ 
tion  contained  in  the  System  Activation  Plan  and,  in  effect,  replace  it  with 
the  following  documents: 

(1)  Installation  and  Checkout  Plan. 

(2)  Maintenance,  transportation,  packaging, and  logistic  support  require¬ 
ments. 

Early  in  the  Acquisition  Phase,  the  SPO  prepares  a  Site  Activation  Plan  which 
serves  as  the  basic  management  document  for  all  activities  relating  to 
activation  at  the  Category  II  test  site  and  at  each  subsequent  site  of  a 
multi-site  system.  The  Site  Activation  Plan  is  based  upon  and  responsive  to 
the  requirements  contained  in  the  Category  I  Test  Plans,  the  Category  II 
Test  Plan,  Installation  and  Checkout  Plans,  and  the  Implementation  Test  Plan. 

Applicable  Data  Item 


T-101  System  Test  Plan 


D-U.  Provide  Inputs  to  the  Request  for  Proposal  (RFP) 


The  RFP  represents  a  condensed  and  organized  product  of  the  actions  of  all 
functional  elements  of  the  SPO  up  to  this  point  in  the  system  program. 
Significant  system  engineering  inputs  comprise  the  principal  content  of  such 
basic  portions  of  the  RFP  as  the  System  Specification,  Statement  of  Work  for 
Phase  B,  Specimen  Work  Statement  for  Development,  System  Test  Plan,  and  Back- 
Up  Data  Package.  Additionally,  program  planning  information  in  the  form  of 
initial  plans  and  schedules,  the  initial  program  breakdown  structure,  and 
program  management ‘network  depend  in  varying  degrees  on  system  engineering 
for  the  work  content  on  which  plans  are  based. 

In  the  system  engineering  area,  the  Phase  B  Work  Statement  will  typically 
contain  a  series  of  paragraphs  which  define  requirements  for  contractor 
accomplishment  of  system  engineering  tasks  in  accordance  with  AFSCM  375-5, 
including  the  required  adaptation  of  Exhibit  2  requirements  for  controls  and 
documentation  to  fit  the  given  Phase  B  program.  The  content  of  this  section 
of  the  Work  Statement  is  based  on  the  SEIP  previously  submitted  as  part  of 
the  PTDP  (see  Block  C-6).  It  will  also  cite  specific  trade  studies  to  be 
conducted,  together  with  requirements  for  preparing  a  Part  I  Detail  Specifica¬ 
tion  for  each  CEI,  expanding  the  System  Specification  (see  Block  D-2),  and 
accomplishing  personnel  subsystem  tasks  associated  with  each  system  segment. 
For  the  computer  program  segment,  tasks  and  contractor  data  products  should 
reflect  an  appropriate  adaptation  of  elements  described  under  Blocks  D-7 
through  D-33  below. 
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D-5.  Prepare  Contractor  Proposals 


While  this  step  will  normally  have  been  preceded  by  varying  degrees  of  contractor 
effort  associated  with  conceptual,  advanced  development,  feasibility,  and/or 
independent  research  and  development  activities  for  the  system,  issuance  of  the 
RFP  initiates  a  series  of  concerted  actions  to  review,  evaluate,  and  expand  the 
System  Specification,  establish  a  proposed  design  approach,  and  organize  a 
continuing  preliminary  design  effort. 

The  System  Specification,  together  with  Functional  Diagrams,  RASs,  Schematic 
Block  Diagrams ,  and  associated  system  engineering  data  resulting  from  earlier 
studies  (which  are  included  with  back-up  data  contained  in  the  RFP)  provide  the 
initial  basis  for  critical  review,  expansion,  and  modification  to  reflect  the 
contractor's  experience  and  planned  approach  to  meeting  system  requirements. 
Additional  trade  study  reports  and/or  time-line  analyses  may  be  generated  in 
the  proposal  preparation  process,  as  well  as  other  appropriate  system 
engineering  documentation  to  support  the  identification  of  major  CEI ,  facility, 
and  gross  personnel  requirements.  These  activities  lead  to  a  first  level  of 
expansion  of  the  System  Specification,  possibly  with  proposed  modifications  at 
this  time,  together  with  expanded  and  modified  system  engineering  data  to  be 
included  with  the  contractor's  proposal.  Expansions  of  the  System  Specification 
normally  relate  primarily  to  requirements  allocations  by  system  segments, 
including  inter-segment  interfaces  (see  Block  D-2).  In  the  case  of  an  associate 
contractor  structure,  each  associate  reponds  to  the  segment(s)  for  which  he  is 
responsible,  as  well  as  to  the  pertinent  general  requirements  contained  in 
other  sections  of  the  System  Specification. 

Applicable  Data  Item 

S-5-l^.O  Technical  Proposal 

Two  proposals  are  required  for  each  contractor,  one  a  firm  proposal 
for  the  Phase  B  effort  and  the  other  a  planning  proposal  for  development  in 
response  to  the  specimen  development  SOW. 

For  the  computer  program  system  segment,  it  is  to  be  noted  that 
the  entire  acquisition  effort,  including  necessary  duplication  (i.e.,  "pro¬ 
duction")  of  computer  programs,  will  be  covered  by  the  ensuing  development 
contract  to  be  awarded  at  the  outset  of  the  Acquisition  Phase.  In  general, 
proposals  for  the  computer  program  segment  will  not  contain  discussions  in  such 
areas  as  reliability,  maintainability,  production,  environmental  testing, 
and  similar  equipment-oriented  concepts.  These  deviations  should  be  noted 
on  the  Form  9  back-up  sheets.  Emphasis  should  be  placed  on  activities  and 
products  which  are  discussed  below  in  the  Definition  and  Acquisition  Phase 
chapters  of  this  guide. 


PHASE  B  -  CONTRACTOR  DEFINITION 


D-6.  Award  Contract(s) 

Following  the  completion  and  submission  of  contractor  proposals,  evaluation 
and  source  selection  activities  result  in  selection  of  the  contractors 
who  will  accomplish  Phase  B.  A  significant  SPO  activity  at  this  time 
is  to  revise  and  update  the  Definition  Phase  Plan  to  incorporate  detailed 
integration  and  implementation  procedures  tailored  to  the  system  program  and 
contractor  structure  selected  (e.g.,  IAC,  associate  contractor,  or  teams). 

The  plain  will  include  a  completed  SEIP  (see  Block  C-6)  reflecting  the  Phase  B 
and  Development  Specimen  Work  Statements  which  have  resulted  from  contractor 
proposals  aind  the  completion  of  contract  negotiations. 

The  contracts  awairded  at  this  time  initiate  the  start  of  contractor  technical 
efforts  in  depth  to  define  the  total  requirements  of  the  system.  From  this 
point,  the  system  program  proceeds  on  contractual  schedules  towards  definition 
of  the  design  requirements  baseline  and  associated  plans  which  will  scope  and 
pace  the  total  technical  effort  for  the  Acquisition  Phase. 
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D-7.  Establish  and  Maintain  System  Documentation  File 


The  purpose  of  this  activity  is  to  provide  a  common  base  of  authoritative  tech¬ 
nical  information  for  use  by  contractor  personnel  in  carrying  out  the  various 
lines  of  system  engineering  effort  required  during  Phase  B.  Primary  objectives 
are  to  insure  that  consistent  and  up-to-date  interrelations  are  maintained 
both  among  the  Phase  B  technical  products  within  the  information  processing 
area  and  with  other  elements  of  the  system.  Normally,  these  objectives  can 
best  be  met  by  establishing  a  central  file  of  the  system  engineering  data  which 
exist  at  the  outset  of  Phase  B  and  instituting  systematic  procedures  for 
validating,  updating,  utilizing,  and  expanding  the  file  as  new  or  revised  data 
are  developed  in  the  course  of  Definition  Phase  activities. 

Initially,  the  file  should  contain  such  formal  documents  as  the  System 
Specification  and  System  Test  Plan,  as  well  as  available  system  engineering 
analyses,  functional  diagrams,  feasibility  and  trade  study  reports,  etc. 
generated  during  the  Conceptual  Phase.  It  may  also  include  available  data 
relating  to:  training,  operations,  and  maintenance  concepts;  Using  Command 
organization;  interfacing  systems;  and  specifications  or  other  descriptions  of 
existing  and  proposed  equipment,  facilities,  communications,  and  computer  pro¬ 
grams  associated  with  the  system.  Expansion  and  revision  of  the  data  occur 
throughout  Phase  B  as  new  information  results  both  from  the  contractor's  tech¬ 
nical  analyses  and  from  external  sources. 

When  properly  organized  and  managed,  the  file  constitutes  both  a  source  and  a 
repository  of  technical  data  relating  to  computer  programs  and  personnel,  and 
to  the  interrelations  of  these  with  other  system  elements.  As  such,  it 
provides  a  focal  point  for  validating  the  available  information  with  system 
requirements,  for  insuring  that  a  common  base  of  technical  requirements  and 
constraints  is  available  for  utilization  in  the  different  lines  of  effort  within 
the  contractor's  organization,  and  for  verifying  technical  integration  of  the 
contractor's  Phase  B  products. 

The  concept  and  objectives  of  the  file  as  described  are  related  to  and 
include  those  of  the  Personnel-Equipment  Data  (PED)  element  of  the  Personnel 
Subsystem  as  set  forth  in  AFR  30-8  and  AFSCM  80-3,  but  are  extended  here  to 
encompass  system  functions  and  tasks  to  be  performed  by  computer  programs  as 
well  as  those  performed  by  personnel.  At  the  same  time,  the  PED  portion 
represents  a  special  subset  of  PED  for  the  system  as  a  whole,  in  that  primary 
concern  in  this  system  segment  is  with  those  operational  and  supporting  personnel 
for  whom  PS  data  in  the  areas  of  human  engineering,  QQPRI ,  training,  technical 
manuals,  and  PSTE  are  dependent  upon,  and  require  close  integration  with,  the 
design  of  computer  programs.  Typically,  these  are  personnel  in  the  operational 
(i.e.,  information  system/subsystem),  simulation/exercising,  and  computer 
programming  support  areas . 


D-8.  Expand  Operational  Functions 


This  activity,  in  conjunction  with  the  activity  described  in  Block  D-9,  is 
concerned  with  expanding  system  functions  allocated  to  the  information  pro¬ 
cessing  system  segment,  detailing  associated  performance/design  requirements, 
and  determining  suitable  implementation  methods,  in  general  terms.  System 
functions  were  initially  identified  at  a  gross  level  and  tentatively  allocated 
to  system  segments  during  Conceptual  Phase  studies.  Additional  analyses  were 
conducted  by  the  SPO  during  Definition  Phase  A  to  expand  system  functions, 
identify  major  performance /design  requirements,  and  verify  allocations  to 
system  segments.  Thus,  the  activity  described  here  is  another  iteration  of  a 
process  which  began  in  the  Conceptual  Phase  and  which  will  continue  throughout 
the  Definition  Phase  B.  Each  iteration  derives  the  additional  level  of  detail 
required  to  proceed  to  the  next  step  of  the  system  design  and  development. 

Functions  allocated  to  the  information  processing  system  segment  may  normally 
be  classified  as  either  operational,  or  supporting.  Operational  functions,  as 
discussed  here,  consist  of  those  functions  which  are  required  directly  and 
immediately  to  carry  out  the  operational  mission(s)  of  the  system.  Supporting 
functions,  discussed  in  Block  D-9,  are  necessary  and  important  for  development, 
test,  and  other  supporting  roles,  but  are  not  directly  required  for  performance 
of  the  system  operational  mission. 

The  level  of  detail  contained  in  the  descriptions  of  the  functions  available 
at  this  time  will  vary  widely  from  system  to  system,  and  within  a  given  system, 
from  function  to  function.  The  System  Specification  for  a  new  system  that  is 
very  similar  to  one  already  in  the  operational  inventory  may  contain  sufficient 
detail  that  the  only  activity  required  at  this  point  would  be  to  review  and 
verify  the  descriptions  and  allocations  of  functions,  subfunctions,  etc.  In 
the  more  typical  case,  however,  the  information  available  at  this  time  will 
consist  only  of  gross  descriptions  of  system  functions,  accompanied  in  some 
instances  by  system-level  design  requirements  and  constraints. 

The  major  effort  required  at  this  time  is  to  break  down  the  gross  functions  into 
lower  levels,  i.e.,  into  subfunctions,  sub-sub functions ,  etc.,  and  determine 
at  each  level  the  associated  performance/design  requirements.  In  this  process 
it  will  be  necessary  to  consider  (l)  system  operational  concepts,  (2)  design 
constraints  imposed  by  the  System  Specification,  (3)  probable  operational 
environment,  (h)  requirements  for  input  data,  processing,  and  output  data,and 
(5)  personnel  capabilities  and  limitations. 

The  initial  objective  of  this  analysis  is  to  reach  the  level  of  detail  required 
to  (l)  establish  the  basis  for  allocating  the  implementation  of  design  require¬ 
ments  among  operational  computer  programs,  manual  operations,  or  joint  man/ 
machine  operations,  and  (2)  adequately  define  interrelationships  of  functions/ 
subfunctions  to  permit  subsequent  man/machine  task  analyses  (Block  D-12), 
further  definition  and  analysis  of  computer  program  tasks  (Block  D-ll),  and 
analyses  of  personnel  requirements  (Block  D-12). 
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D-9.  Expand  Supporting  Functions 


The  activity  referred  to  in  this  block  is  comparable  in  some  respects  to  that 
described  in  the  preceding  Block  D-8,  but  is  concerned  with  a  variety  of  non¬ 
operations  functions  which  involve  requirements  for  computer  programs.  These 
"supporting"  items  are  sub-classified  as  utility,  maintenance-diagnostic, 
and  support  (see  explanation  of  terms  in  paragraph  A, 4  at  the  beginning  of 
this  chapter).  While  they  represent  functions  which  may  or  may  not  correspond 
with  separate  CPCEIs,  they  are  referred  to  herein  as  separate  CPCEIs  for  con¬ 
venience  of  discussion. 

It  is  to  be  assumed  that  the  requirements  for  these  functions  will  have  been 
established  at  varied  levels  of  specificity  prior  to  this  time,  and  that  the 
relative  emphasis  on  utility,  maintenance-diagnostic,  and  support  elements  may 
vary  widely  for  different  systems.  In  general,  the  approaches  to  developing 
Part  I  Specifications  for  items  in  these  areas  involve  characteristic  differ¬ 
ences  in  required  phasing,  types  of  analysis,  and  degree  of  dependence  upon 
other  Phase  B  activities  concerned  with  operations. 

For  example,  major  functions  to  be  considered  in  the  utility  area  may  include: 
compiling/assembling  requirements ,  for  translating  computer  programmer 
language  into  machine  input  code;  tape  loading  and  maintenance,  for  generating, 
manipulating,  cataloging,  duplicating,  verifying,  and  interpreting  computer 
program  tapes;  requirements  for  test/debugging  of  tape  contents;  and  data 
description  requirements,  e.g.,  for  a  Communication  Pool  (COMPOOL)  and 
associated  COMPOOL  generating  and  analysis  tools. 

In  large  part ,  the  performance /design  requirements  for  these  functions  are 
typically  derived  from  and  associated  with  design  characteristics  of  the 
computer  equipment.  For  a  given  computer,  most  of  the  resulting  utility  tools 
are  useful  for  any  system  or  non-system  data  processing  applications  for  which 
the  computer  might  be  employed.  Many  of  these  tools  are  often  supplied  with 
the  computer.  However,  it  is  also  often  true  that  additional  or  special  utility 
elements  must  be  designed  to  take  account  of  specific  uses  or  general  design 
characteristics  of  the  operational  computer  program  elements  in  a  given  system; 
the  latter  is  typically  true  of  the  COMPOOL,  for  example,  which  is  structured 
to  satisfy  data  base  requirements  of  the  given  system,  within  storage  con¬ 
straints  of  the  computer. 

Design  requirements  in  the  maintenance-diagnostic  area  are  also  derived  basically 
from  design  characteristics  of  the  computer.  The  maintenance-diagnostic  computer 
programs  may  perform  such  functions  as  detection  of  inoperative  circuits,  re¬ 
routing  of  computer  functions  through  redundant  circuitry,  and  display  of 
information  to  assist  a  monitoring  operator  to  evaluate  the  status  of  computer 
functioning.  These  computer  programs  may  have  significant  operating  inter¬ 
faces  with  the  operational  computer  programs,  either  through  time  sharing  or 
on  a  near  real-time  basis.  Typically,  they  represent  important  elements  of, 
or  contributors  to,  the  computer  equipment  reliability. 


33 


Support  computer  program  requirements  are  derived  from  analyses  of  specific 
needs,  characteristics,  or  intended  utilization  of  the  system  personnel  and 
operational  computer  programs.  They  are  intended  for  Operational  Phase  use, 
primarily,  but  may  also  be  required  during  Acquisition,  for  such  purposes  as 
test,  evaluation,  and  personnel  training.  Major  functions  performed  by  these 
computer  programs  are  typically  in  the  areas  of  (a)  simulation  and  (b)  record¬ 
ing/data  reduction.  Simulation  functions  involve  the  generation  of  data,  either 
off-line  or  on  a  dynamic  basis,  to  simulate  inputs  to  or  from  system  equipment, 
personnel,  or  computer  programs.  Data  recording  and  reduction  encompass  such 
functions  as  selecting,  recording,  storing,  sorting,  analyzing,  reducing,  and 
displaying  data  which  are  generated  or  manipulated  by  the  operational  computer 
programs . 

Hence,  activities  initiated  at  this  time  will  vary  in  depth  and  level  as  a 
function  of  the  above  considerations.  In  the  utility  area,  analysis  of  functions 
and  definition  of  design  requirements  can  proceed  rapidly  and  in  depth  at  an 
early  stage.  Some  utility  tools  which  will  be  needed  during  early  Acquisition 
for  developing  other  computer  programs  may  require  not  only  completed  Part  I 
Specifications  but  also  completed  preliminary  or  detail  design  prior  to  the 
outset  of  Acquisition;  others  will  require  essential  design  inputs  from  the 
operational  CPCEI  analysis  activities  (e.g.,  Block  D-20).  Expansion  and 
definition  of  support  functions  may  be  initiated  on  a  limited  basis  at  this 
time,  but  will  depend  closely  on  essential  inputs  generated  during  the  course 
of  Phase  B  in  the  system  exercising,  test,  and  operational  CPCEI  analysis/ 
design  activities. 
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D-10.  Define  User's  Operating  Organization  and  Allocate  Personnel  Functions 


This  activity  is  directed  towards  defining  the  structure  of  personnel  for  the 
using  agency's  conduct  of  system  operations,  and  allocating  system  functions 
to  identified  types  of  operating  personnel  and/or  duty  stations.  It  is 
initiated  on  the  basis  of  information  made  available  to  the  contractor  at 
the  outset  of  Phase  B  relating  to  the  user's  operating  organization,  and  is 
extended  at  this  stage  as  a  continuation  of  the  functional  analysis  and 
allocation  activities  conducted  in  the  preceding  Block  D-8. 

The  identified  functions  to  be  performed  by  operational  personnel  will  generally 
fall  into  two  broad  categories,  namely: 

(1)  Manual  —  functions  which  do  not  imply  direct  interaction  with 
the  computer,  but  which  are  essential  to  the  operational  mission. 

These  will  include  decision,  planning,  coordinating,  communica¬ 
tion,  recording,  status  posting,  etc.  functions  performed  by 
command  and/or  staff  personnel  of  the  operating  organization. 

(2)  Man-Machine  —  those  functions  which  are  directly  associated  with 
computer  operations,  to  be  performed  by  personnel  located  at,  or 
having  access  to,  consoles,  displays,  or  other  input/output 
equipment. 

These  functions  are  grouped  into  duty  positions,  taking  into  account  such 
factors  as  (l)  physical  locations  at  which  the  functions  must  be  accomplished, 

(2)  levels  of  skills  and  knowledge  required,  (3)  frequencies  and  timing 
requirements,  (^)  communication  demands,  (5)  accuracy  and  reliability  require¬ 
ments,  and  (6)  alternate  modes  of  system  operation. 

This  activity  represents  an  iteration  and  refinement  of  studies  accomplished 
during  Conceptual  Transition,  utilizing  the  more  detailed  definitions  of 
functions  available  at  this  time,  together  with  the  personnel  and  equipment 
concepts,  constraints,  and  design  requirements  dictated  by  the  System  Specifica¬ 
tion.  Account  is  taken  of  available  information  relating  to  training, 
operation,  and  maintenance  concepts  established  in  the  PTDP;  and  additional 
contractor  effort  is  typically  required  to  collect  further  detailed  data 
relating  to  Using  Command  mission,  organization,  doctrine,  and  operating 
procedures . 

Further  refinements  should  occur  throughout  the  remainder  of  Phase  B.  The 
result  at  this  stage  is  an  interim  set  of  lists  of  personnel  functions,  grouped 
by  types  of  operating  position,  which  provide  working  documents  for  subsequent 
analysis,  revision,  and  expansion  in  the  course  of  (l)  developing  human  engineer¬ 
ing  design  recommmendations  for  equipment,  communications,  workspaces,  and 
computer  programs  (Block  D-12),  and  (2)  providing,  under  the  QQPRI  activity,  a 
complete  forecast  of  operational  organization  and  personnel  composition. 
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Subsequent  activity  will  also  normally  include  the  development  of  similar 
information  for  personnel/organizational  requirements  in  the  areas  of 
simulation/exercising  and  computer  programming  support  (Blocks  D-13  and  D-25). 
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D-ll.  Define  Operational  Computer  Program  Tasks 


Based  upon  preceding  identification  of  major  functions  to  be  performed  by  the 
operational  computer  program  (Block  D-8),  this  activity  undertakes  the  further 
analysis  and  breakdown  of  these  functions  into  subfunctions  and  tasks,  and  to 
initiate  the  determination  of  performance/design  requirements  at  each  level. 

Broadly,  the  individual  tasks  to  be  performed  will  be  either  in  the  nature  of 
logical  manipulations  of  data,  or  mathematical  computations.  Objectives  are 
to  define  the  tasks ,  define  and  evaluate  their  interrelationships ,  and  progress 
towards  the  formulation  of  comprehensive  rules  to  govern  each  computation 
and  logical  manipulation. 

The  process  should  typically  involve  the  identification  of  alternative  sets  of 
subfunctions  and  tasks  for  meeting  the  system  operating  requirements,  as  well 
as  alternative  rules  or  computational  techniques  to  be  employed  for  accomplish¬ 
ing  the  individual  tasks.  Since  the  alternatives  are  typically  numerous,  trade¬ 
off  studies  in  depth  should  be  accomplished  only  for  those  sets  of  alternatives 
which  offer  significant  payoffs  in  relation  to  system  objectives.  Selections 
are  based  on  established  system  requirements,  design  feasibility  and  efficiency, 
and  impact  on  other  system  elements. 

For  each  task,  it  is  necessary  to  identify  the  source  and  form  of  input  data, 
initial  operations  performed  (e.g.,  quality  monitoring),  all  logical  manipula¬ 
tions  and  computations  to  be  accomplished,  and  relevant  output  interfaces 
with  other  tasks,  together  with  alternative  modes,  conditions,  and  rules  of 
operation.  The  definition  of  sequencing  and  interactions  among  tasks  will  be 
initiated  at  this  time,  leading  to  the  specification  of  requirements  for 
executive  control  functions. 

Although  this  activity  is  based  upon  a  previously-established  general  frame¬ 
work  of  functional  requirements  allocated  to  the  operational  computer  programs 
(Block  D-8) ,  the  studies-in-depth  may  result  in  some  recommended  re-allocation 
of  functions.  In  addition  to  progress  towards  development  of  Part  I 
Specification s)  for  operational  CPCEI  functions,  it  may  normally  also  result 
in  detailed  requirements  for  man-machine  functions  (Block  D-12),  detailed 
equipment  characteristics  (Block  D-27),  and  requirements  pertaining  to 
supporting  computer  programs  (Block  D-l8,  D-22). 
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D-12.  Conduct  Man-Machine  Task  Analysis 

This  activity  consists  of  detailed  expansion  and  analysis  of  human  operator 
functions  tentatively  assigned  under  the  preceding  Block  D-ll.  While  it  also 
initiates  the  development  of  procedural  data  which  will  have  subsequent  uses 
in  such  areas  as  personnel,  training,  and  handbooks,  its  primary  purpose  at 
this  stage  is  to  develop  and  verify  recommended  performance/design  requirements 
for  interfacing  computer  programs  and  equipment. 

Specific  approaches,  degree  of  detail,  and  emphasis  should  vary  as  a  function 
of  such  factors  as  established  constraints,  novelty,  and  criticality  of 
individual  tasks  to  system  performance.  However,  the  preliminary  grouping 
of  tasks  into  operating  personnel  positions  (Block  D-10)  is  an  essential  pre¬ 
requisite  to  the  analysis,  since  the  degree  to  which  human  operators  will 
meet,  or  fail  to  meet,  required  levels  of  performance  on  individual  tasks 
is  typically  a  complex  function  of  the  total  job  demands  and  working  environment. 
Where  indicated,  the  initial  approach  may  be  based  upon  the  use  of  preliminary 
drawings  or  a  mockup  of  the  operating  station  to  devise  detailed  human  action 
sequences  for  each  task,  identifying  (l)  specific  initial  conditions,  (2)  infor¬ 
mation  required,  (3)  information  available,  (4)  required  evaluations/ 
decisions/coordination,  (5)  required  actions,  frequencies,  tolerances,  and 
available  controls,  and  (6)  nature  of  feedback. 

Supplementary  analyses  or  studies  may  be  required  to  verify  timing,  workloads, 
consistency  or  complexity  of  display  formats,  appropriate  control  actions  and 
control  response  characteristics,  and  logical  control-display  relationships. 

For  critical  functions,  or  where  significant  cost  alternatives  exist,  special 
experiments  or  trade-off  studies  may  be  required  to  identify  and  evaluate 
feasible  design  alternatives. 

The  objective  of  this  activity  is  to  specify  performance  and  design  character¬ 
istics  of  computer  programs  and  associated  equipment  which  will  be  consistent 
with  natural  capabilities  and  limitations  of  human  operators,  in  the  interests 
of  minimizing  requirements  for  special  personnel  selection  or  training  and 
of  promoting  efficient  operation  of  the  system  as  a  whole.  The  principal  direct 
output  consists  of  detailed  definitions  of  content  and  formats  of  displays, 
varieties  and  sequences  of  switch  or  other  manual  input  actions,  and  the  nature 
and  timing  of  computer  response  to  operator  interventions,  including  require¬ 
ments  for  modified  and/or  special  automatic  data  processing  functions. 

Detailed  performance/design  requirements  for  these  manual  interfacing  functions 
are  derived  in  coordination  with  deriving  other  performance/design  requirements 
for  operational  computer  programs  (Block  D-ll),  taking  into  account  constraints 
dictated  by  considerations  of  feasibility  and  integration  of  requirements  for 
the  CPCEl(s)  as  a  whole. 

The  responsibility  for  these  man-machine  task  analyses  and  development  of  pro¬ 
cedural  data  for  system  operators  is  normally  associated  with  this  segment, 
because  computer  programs  are  the  major  determiners  of  specific  nmachine,! 


43 


functions  in  an  information  processing  system.  At  the  same  time,  the  activity 
also  requires  the  development  of  certain  associated  design  requirements  for 
equipment,  facilities,  and  communications.  For  example,  details  of  available 
display  elements,  switch  or  pushbutton  numbers  and  arrangements,  direct  or 
remote  communication  with  other  system  personnel,  etc.  must  be  known  or 
devised,  and  evaluated  in  the  course  of  each  task  analysis.  Hence,  this  effort 
should  be  planned  as  a  primary  source  of  human  engineering  design  requirements 
in  those  areas.  Normally,  the  recommendations  should  constitute  expansions 
in  detail  only,  within  the  scope  of  general  requirements  which  have  been  antici¬ 
pated  and  established  in  the  System  Specification.  Depending  on  the  contractor 
structure  in  a  given  Phase  B  program,  some  of  the  recommended  performance/design 
requirements  for  CEIs  in  other  system  segments  may  be  input  immediately  for 
review  and  coordination  by  the  responsible  contractor ( s) .  They  are  normally 
combined  with  similar  recommendations  derived  from  other  Phase  B  activities 
(e.g.,  Blocks  D-21,  D-2J+),  summarized  as  a  special  section  of  the  final 
report  (Block  D-33),  and  appropriately  reflected  in  interface  sections  of  the 
Part  I  CPCEI  Specification s ) .  Following  review,  coordination,  and  approval, 
they  should  be  subsequently  incorporated  into  Part  I  Specifications  of  the 
affected  equipment/facility  CEIs,  usually  prior  to  PDRs  for  those  items. 

As  implied  above,  the  human  engineering  recommendations  resulting  from  this 
activity  are  at  the  performance/design  requirements  level  only.  For  equip¬ 
ment  CEIs,  for  example,  they  might  consist  of  specifying  such  items  as: 
numbers  and  types  of  display  characters  available;  numbers,  arrangement, 
functions,  and  labels  for  console  pushbuttons;  numbers  and  types  of  available 
warning  lights  or  auditory  alarms.  For  these  CEIs,  additional  human  engineer¬ 
ing  design  is  normally  required  in  the  course  of  subsequent  detail  engineering 
design  leading  to  the  Part  II  specifications,  both  for  operability  and  main¬ 
tainability,  including  many  of  the  factors  treated  in  MIL-STD-803A  in  the  areas 
of  anthropometry,  illumination  and  visibility,  ambient  noise  levels,  and 
control-display  sizes,  shapes,  friction,  inertia,  etc. 

For  computer  program  CEIs,  since  they  are  inherently  functional  items,  factors 
affecting  interfaces  with  human  performance  during  computer  program  operation 
must  be  completely  specified  at  the  performance/design  requirements  level, 
i.e.,  in  the  Part  I  Specification.  Generally,  these  considerations  should  be 
summarized  in  Section  3.1.^,  "Human  Performance  Requirements",  to  provide 
an  explicit  basis  for  Personnel  Subsystem  Test  and  Evaluation  (PSTE)  during 
Category  I  testing.  However,  since  they  are  necessarily  incorporated  in  the 
basic  requirements  contained  throughout  Section  3.0  of  the  specification,  they 
do  not  normally  imply  additional  requirements  for  the  computer  programming 
design  and  development  effort  during  Acquisition. 

Interrelationships .  The  analysis  of  man-machine  tasks,  as  indicated  above, 
is  accomplished  at  this  stage  primarily  for  the  purpose  of  identifying  and 
evaluating  performance/design  characteristics  of  computer  programs  and 
equipment.  While  it  continues  during  Phase  B  as  a  support  activity  to  the 
development  of  Part  I  CPCEI  Specifications,  the  task  data  are  progressively 
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refined,  organized,  and  expanded  into  sets  of  procedural  data  (Block  D-29)  which 
will  provide  the  basis  for  training  manuals,  positional  handbooks,  and  other 
Job  aids  to  be  developed  subsequently  in  the  Acquisition  Phase. 

At  this  stage,  the  provisional  task  data  are  also  input  to  the  QQPRI  activity 
(Blocks  D-13  to  D-30)  for  further  development  of  personnel-oriented  position 
descriptions,  classification,  and  identification  of  individual  training  require¬ 
ments.  Additionally,  the  data  provide  the  basic  material  for  analyzing 
operational  readiness  training  needs  and  developing  functional  requirements 
for  operational  system  exercising  (Block  D-l6). 


D-13 .  Initiate  Analysis  of  Personnel  Requirements 


In  Block  D-10  preceding,  manual  and  man-machine  tasks  were  identified  and 
grouped  into  duty  positions  on  the  basis  of  preliminary  estimates  of  the 
characteristics  of  the  tasks,  user  organization  and  mission,  and  general 
guidance  contained  in  the  System  Specification.  Using  those  results  as  an 
input,  this  activity  initiates  a  continuing  series  of  analyses  oriented  toward 
identifying,  by  Air  Force  Specialty  Code,  the  numbers  and  types  of  Air  Force 
Personnel  which  will  be  required  to  operate  the  system.  The  activity  is  the 
primary  source  of  data  for  the  preparation  of  the  QQPRI  report,  initially 

prepared  at  the  end  of  Definition  Phase  B  (see  Block  D-30),  and  revised 
and  updated  during  the  Acquisition  Phase  (see  Block  A-l6). 

Initially,  the  activity  should  expand  the  duty  position  descriptions  identified 
in  Block  D-10  and  derive  preliminary  estimates  of  operator  qualifications 
for  each  position.  Qualifications  would  include  such  considerations  as  rank, 
primary  and  secondary  skills  relevant  to  selection,  additional  skills  and 
knowledge  to  be  acquired  in  training,  and  expected  duration  of  training  status. 
To  this  end,  close  coordination  must  be  maintained  with  the  man-machine  task 
analysis  described  in  Block  D-12  which  provides  the  major  source  of  informa¬ 
tion  and  data  refining  duty  position  descriptions  and  defining  operator 
qualifications.  In  addition,  descriptions  of  duty  positions  for  computer  pro¬ 
gramming  support  personnel  are  initiated  at  this  time.  This  work  will 
involve  a  detailed  consideration  of  the  various  facets  of  computer  programm¬ 
ing  support  which  will  be  required.  Some  of  these  relevant  facets  include: 

(l)  on-site  versus  off-site  computer  program  design,  coding,  and  test 
support;  (2)  provisions  for  control  of  changes;  (3)  the  frequency  and  extent 
of  major  changes;  and  (U)  the  varying  levels  of  computer  programming  support 
to  the  military  organization,  e.g.  ,  advisory,  line,  staff,  operations. 

As  a  clearer  picture  emerges  of  the  various  duty  positions  and  associated 
operator  qualifications,  it  must  be  examined  in  relation  to  the  userfs  (l) 
mission,  doctrine,  and  operating  procedures,  and  (2)  personnel  organization  and 
classification  structure.  At  this  point  in  the  analysis,  sufficient  detail 
should  be  available  to  allow  comparison  of  each  duty  position  and  associated 
operator  qualifications  with  existing  Air  Force  Specialties  (AFS)  and  the 
identification  of  the  appropriate  Air  Force  Specialty  Code  (AFSC)  for  each 
position.  If  new  AFSs  or  shredouts  of  existing  AFSs  are  required  they  should 
be  identified  at  this  point  and  analyses  initiated  to  determine  what  training 
will  be  required  to  qualify  personnel  in  the  new  AFS. 

As  a  final  step  in  the  analyses,  attention  should  be  directed  toward  pertinent 
manning  factors  such  as  workloads,  special  working  conditions,  number  of  shifts, 
etc.,  to  derive  preliminary  estimates  of  numbers  of  personnel  required  for 
each  duty  position.  Operational  and  maintenance  concepts  for  the  system  as 
documented  in  the  PTDP  and  the  System  Specification  should  provide  basic 
guidance  in  these  analyses. 


k6 


D-lU.  Initiate  Detailed  Test  Planning 


This  activity  initiates  the  contractor’s  planning  for  accomplishing  develop¬ 
mental  testing.  The  System  Test  Plan,  prepared  by  the  SPO  and  reviewed  by 
the  contractor  during  Definition  Phase  A,  is  the  basic  guidance  document  for 
the  activity.  It  contains  overall  test  philosophy,  basic  concepts  and 
objectives  for  Category  I  and  system  tests,  and  rudimentary  planning  informa¬ 
tion.  The  planning  activity  initiated  at  this  time  and  continued  throughout 
the  development  cycle  represents  a  progressive  refinement  and  expansion  of 
the  material  initially  contained  in  the  System  Test  Plan.  At  the  end  of  the 
Definition  Phase  B,  the  System  Test  Plan  is  replaced  by  a  preliminary  Category 
I  Test  Plan  and  inputs  to  the  Category  II  Test  Plan  (Block  D-32).  During 
the  Acquisition  Phase,  these  planning  documents  are  expanded,  detailed  pro¬ 
cedures  prepared,  tests  conducted,  data  analyzed*  and  reports  prepared. 

At  this  point  in  time  it  is  necessary  to  expand  the  basic  cest  concepts  and 
objectives  with  reference  to  the  contractor’s  particular  approach  to  the 
development  process,  integrate  these  concepts  and  objectives  into  the  design 
process  and  develop  preliminary  plans  for  implementing  the  actual  testing 
activity.  For  example,  the  System  Test  Plan  should  present  the  basic  philosophy 
of  qualification  testing  of  computer  programs  and  discuss  the  concepts  and 
objectives  of  Preliminary  and  Formal  Qualification  Testing.  The  contractor 
should  refine  these  in  light  of  his  plan  and  organization  for  accomplishing 
CPCEI  design,  and  insure  that  design  personnel  are  familiar  with  the  concepts 
and  objectives  and  integrate  them  into  their  design  effort.  Such  integration 
of  test  concepts  and  objectives  into  the  design  process  is  important  for  two 
reasons : 

(1)  Test  concepts  and  objectives  may  serve  as  design  constraints  in  that 
they  establish  the  basic  framework  for  testing  within  which  all  design  solutions 
must  be  testable. 

(2)  Detailed  test  requirements  generated  by  the  design  process  should  be 
organized  and  presented  in  Section  U  of  the  Part  I  Detail  Specification  in  a 
manner  consistent  with  the  test  concepts  and  objectives. 

As  design  of  the  operational  computer  programs  evolves,  analyses  should  be 
conducted  to  identify  those  test  requirements  which  affect  the  supporting 
computer  programs.  Significant  effects  may  occur  in  terms  of  requirements 
for  specific  capabilities  which  the  supporting  CPCEIs  must  provide  »  and/or  in 
terms  of  forcing  their  development  schedule  to  meet  specific  deadlines  in 
testing  the  operational  CPCEIs.  These  requirements  will  be  identified 
initially  at  gross  levels,  but  will  become  progressively  more  detailed  during 
the  Definition  Phase.  By  the  end  of  Phase  B,  they  should  be  reflected  at 
appropriate  levels  in  the  Part  I  Detail  Specifications  and  the  initial  test 
planning  documents  (Blocks  D-26,  D-32). 


The  refinement  of  test  concepts  and  objectives,  the  identification  of  test 
requirements  during  the  design  process,  and  the  determination  of  requirements 
for  supporting  CPCEIs  based  on  operational  CPCEIs  should  be  accompanied 
by  appropriate  planning  for  the  implementation  of  the  test  activities.  Basic 
schedules  tied  to  system  program  milestones  should  be  developed,  appropriate 
test  methods  identified,  controls  established,  responsibilities  assigned,  and 
support  requirements  defined.  The  planning  effort  requires  continuous 
coordination  with  the  technical  design  activity  throughout  Definition  Phase 
B.  It  culminates  in  the  preparation  of  the  Category  I  Test  Plan  and  Inputs 
to  the  Category  II  Test  Plan  (see  Block  D-32)  which  form  part  of  the 
contractors  Phase  B  Final  Report  (see  Block  D-33). 
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D-l 5 •  Define  Detailed  Display/Control  Requirements 


As  a  result  of  analyses  of  functions  accomplished  in  Block  D-ll  and  preceding, 
the  CPCEI  will  have  become  structured  into  an  initial  set,  or  sets,  of 
functional  elements,  e.g.,  information  transfer,  system  limits,  recording, 
assignment,  etc.  Analyses  to  detail  the  subfunctions  and  tasks  within  these 
various  elements,  and  to  identify  and  select  alternative  functional  solutions, 
will  have  been  accomplished  in  the  Block  D-ll  activity  for  tasks  which  are 
identified  at  an  early  point  as  fully  automated  (Block  D-8),  and  in  the 
Block  D-12  activity  for  manual  and  joint  man-machine  functions.  Important 
products  of  the  man-machine  task  analyses  will  be  additional  definitions  of 
detailed  functional  requirements  to  be  met  by  the  CPCEI.  The  purpose  of 
this  activity  is  to  integrate  these  results  into  the  direct  process  of 
developing  the  operational  CPCEI  Part  I  Specification. 

The  specific  requirements  resulting  from  Block  D-12  to  be  incorporated  are 
generally  of  the  three  types  listed  below: 

(1)  Requirements  for  display  formats,  contents,  timipg,  and  other 
relevant  characteristics. 

(2)  Requirements  for  numbers,  positions,  labels,  and  functions  of 
operator  input  devices,  e.g.,  buttons,  switches,  knobs. 

(3)  Requirements  for  special  data  retrieval,  computations,  or  other 
processing  to  support  the  performance  of  operator  duties  and  tasks. 

The  integration  is  accomplished  either  by  adding  functional  elements,  or 
combining  requirements  with  functional  elements  already  identified,  or  both. 
The  process  typically  involves  the  necessity  to  resolve  conflicts  between 
desirable  and  feasible  solutions,  and  often  requires  iterations  of  earlier 
analyses.  The  refinement  and  updating  of  initial  solutions  will  normally 
continue  throughout  the  course  of  CPCEI  Part  I  Specification  development. 

In  determining  the  techniques  to  be  used  in  displaying  information,  considera¬ 
tion  is  given  to  such  items  as  storage  and  timing  in  regard  to  the  computer 
program,  display  capacity,  presentation  rates,  formatting,  display  selection, 
and  symbolic  repertory  characteristics  of  the  display  equipment.  Items  to  be 
defined  in  detail  include: 

(1)  Organization  of  operator  in format ional  exigencies  into  feasible 
groupings . 

(2)  Formats  to  be  used  in  presenting  information. 

(3)  Coding  conventions  to  be  employed  in  presenting  information  details. 

(4)  Rules  for  routing  displays. 
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(5)  Rules  for  forcing  displays, 

(6)  Rules  for  updating  displays. 

(7)  Rules  for  priority  handling  of  displays. 

The  determination  of  ,fuse  rules"  for  manual  intervention  devices  requires 
taking  into  account  the  known  or  recommended  characteristics  of  computer 
and  console  equipment  (see  Block  D-12),  as  well  as  the  sequencing,  frequencies, 
priorities,  etc.  of  information  to  be  input  by  system  operators.  Specific 
information  to  be  derived  will  include: 

(1)  Allocations  of  the  types  of  information  that  can  be  inserted  by  each 
manual  input  device. 

(2)  Frequencies  of  operator  insertion  readout  by  the  computer  program. 

(3)  Informational  structuring  logic  to  be  used  for  keyboard  type  devices. 

(^)  Actuating  rules  for  feedback  devices  which  inform  operating  personnel 
of  the  adequacy  of  their  insertions. 
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D-l6.  Perform  Training  Needs /Exercise  Requirements  Analysis 


In  this  context,  training  needs  refer  to  the  skills  and  knowledge  which  system 
personnel  must  acquire  through  operational  readiness  training  in  order  to 
perform  their  assigned  tasks.  As  such,  they  constitute  the  basis  for  identify¬ 
ing  and  defining  exercise  requirements.  The  analysis  at  this  stage  is  directed 
towards  (l)  identifying  the  special  coordinative  skills  and  knowledges, 
primarily,  which  are  required  of  personnel  in  the  context  of  system  operations, 
and  (2)  specifying,  initially  in  functional  terms,  the  types,  frequencies, 
levels ,  and  conditions  of  system  exercising  which  will  establish  and  maintain 
the  necessary  levels  of  personnel  proficiency. 

Concepts,  requirements,  and  constraints  governing  the  performance  of  this  and 
other  activities  in  the  system  exercising  area  are  derived  from  the  RAD,  Con¬ 
ceptual  Phase  system  engineering  data,  the  PTDP,  and  System  Specification.  During 
Definition  Phase  B,  the  analysis  of  training  needs  should  constitute  an 
iteration,  in  part,  of  earlier  Conceptual  Phase  studies,  using  the  more  detailed 
data  available  at  this  time.  Significant  inputs  are  derived  from  the  grouping 
of  operator  functions  into  duty  positions  (Block  D-10)  and  man-machine  task 
analyses  (Block  D-12).  While  continuity  is  also  required  with  analyses  of 
individual  training  needs  associated  with  QQPRI  (Blocks  D-13,  D-25,  D-30) ,  this 
effort  places  major  emphasis  on  the  skills  which  must  typically  be  built  and 
maintained  thorugh  Operational  Readiness  Training  (ORT).  These  include  the 
pacing,  load-balancing,  information-filtering,  and  decision-making  skills  and 
knowledges  which  are  involved  in  dynamic  interactions  among  individual  crew 
members  and  between  crews  of  separate  organizational  units. 

The  nature  and  depth  of  analysis  required  will  vary  with  the  system,  factors 
of  novelty,  and  established  constraints.  Prerequisite  information  includes 
detailed  data  regarding  the  user's  objectives  and  constraints  pertaining  to 
normal  and  emergency  operations,  alternate  system  modes.  Joint  operation 
with  interfacing  systems  and  agencies,  etc.  Typically,  additional  clarifica¬ 
tion  and  definition  of  user  requirements,  proposed  practice,  and  criteria  must 
be  sought  in  the  course  of  this  activity.  A  systematic  approach  to  the 
analysis  might  include,  as  a  first  step,  assessing  the  identified  interactive 
skills  in  terms  of  criticality,  complexity,  frequency,  probable  loss  in 
proficiency  over  time,  and  potential  for  development  through  practice.  Initial 
objectives  are  to  determine  and  assess  exercising  requirements  in  terms  of 
such  parameters  as  the  following: 

(l)  Exercise  Configuration  —  the  particular  collection  of  personnel, 
operational  equipment,  and  operational  computer  programs  that  are  to  interact 
in  the  execution  of  a  given  exercise.  This  may  vary  in  magnitude  from 
“individual"  through  "subsystem" — i.e.,  functionally  distinct  portion  of  the 
crew  at  one  operation  station— through  successively  larger  organizational 
aggregates  such  as  the  division  or  command,  to  configurations  involving 
other  commands. 
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(2)  Training  Exercise  Objectives  —  the  mission  element  or  collection 

of  mission  elements  to  be  exercised  by  a  given  configuration  of  the  operational 
system  for  purposes  of  developing  and  maintaining  proficiency  of  the  given 
operational  personnel,  including  special  features  and  conditions  of  the  operating 
environment  which  are  to  be  incorporated  in  given  exercise  configurations 
and  missions,  e.g.,  countermeasures,  degraded  communications. 

(3)  Exercising  Conditions  —  conditions  affecting  the  conduct  of  exercises, 
e.g.,  off-station  vs.  on-station  considerations,  time-sharing  requirements, 
simultaneous  simulated  and  live  inputs. 

(k)  Exercise  Frequency  —  the  recommended  frequency  with  which  given 
types  of  exercise  should  be  conducted  to  maintain  an  acceptable  level  of  crew 
or  unit  proficiency.  The  recommended  frequency  for  each  exercise  should 
reflect  the  variety  of  configurations,  missions,  and  conditions,  taking  into 
account  both  uniqueness  and  duplication  among  the  exercises. 

This  activity  should  parallel  the  closely  associated  analysis  of  evaluation 
needs /exercise  requirements  performed  in  Block  D-17-  Specific  implementing 
recommendations  (Block  D-2l)  are  based  upon  combined  functional  requirements 
resulting  from  the  two  analyses,  since  it  is  normally  anticipated  that 
operational  system  exercises  will  be  utilized  for  the  dual  purposes  of  personnel 
training  and  system  evaluation.  The  evaluation  area  will  also  normally  include 
consideration  of  significant  evaluation  requirements  associated  with  personnel 
training  (see  Block  D-17 ) • 

Applicable  Data  Item 

Q-119  Training  Needs /Exercise  Requirements  Analysis. 
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D-17 .  Perform  Evaluation  Needs /Exercise  Requirements  Analysis 


This  activity  is  closely  related  to  the  Training  Needs/Exercise  Requirements 
Analysis  described  in  Block  D-l6,  but  is  directed  specifically  towards 
identifying  the  types  of  exercises  which  are  required  for  purposes  of  evaluation. 
At  this  stage,  it  consists  of  (l)  defining  the  system  functions,  subfunctions, 
and  components  which  require  periodic  test  and  evaluation  during  the  opera¬ 
tional  life  of  the  system  by  the  user  and  (2)  specifying  the  types,  levels,  and 
conditions  of  exercising,  and  of  performance  measurements,  which  are  required 
to  meet  evaluation  objectives. 

Where  Conceptual  Phase  studies  (Block  C-5)  have  resulted  in  establishing  the 
feasibility  and  gross  requirements  for  an  exercising  capability,  it  is  normally 
expected  that  the  capability  will  be  designed  to  meet  a  variety  of  objectives 
in  the  Acquisition  process,  e.g..  Category  I  and  II  testing,  as  well  as  those 
of  Operational  Readiness  Training  (ORT)  and  Operational  Phase  system  evalua¬ 
tion.  While  it  is  also  expected  that  design  requirements  related  to  the  train¬ 
ing  and  evaluation  purposes  will  be  basically  similar  in  general,  specific 
objectives  should  be  examined  at  this  time  in  sufficient  detail  to  assure  that 
necessary  modifications  or  additions  are  not  overlooked.  Typically  the  differ¬ 
ences  between  training  principles  and  evaluation  principles  affect  both  the 
conduct  and  required  technical  support  of  system  exercises  in  significant  ways. 

Major  objectives  and  constraints  are  derived  from  the  System  Specification  and 
other  formal  documents ,  as  well  as  from  Conceptual  Phase  system  engineering 
data.  As  in  the  case  of  training  needs  (Block  D-l6),  additional  clarification 
of  user  requirements  and  criteria  may  also  be  indicated  in  the  course  of 
detailing  evaluation  needs  in  sufficient  depth  to  specify  implementing  solu¬ 
tions.  Purposes  to  be  considered  and  amplified  typically  include  the 
following : 

(1)  Assess  operational  readiness  of  the  system  to  accomplish  its  defined 
missions.  The  focus  of  interest  here  is  on  valid  and  useful  indices  of  total 
system  performance,  together  with  criteria  for  assessing  effectiveness  in 
relation  to  mission  objectives. 

(2)  Assess  alternative  operating  tactics,  techniques,  and  conditions. 

(3)  Determine  criticality  of  individual  functions  and  subfunctions  to 
system  effectiveness. 

(M  Assess  proposed  or  actual  changes  in  system  and/or  component 
configurations. 

(5)  Assess  crew  proficiency,  to  determine  effectiveness  of  training, 
compare  crews,  or  evaluate  and  develop  procedures.  Consideration  should  be 
given  both  to  (a)  typical  needs  of  command  staff  personnel  responsible  for 
evaluation,  who  are  not  necessarily  exercise  participants,  and  (b)  needs  of 
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crew  participants  for  feedback  performance  data  which  can  be  utilized  for  self- 
improvement  . 

Exercise  requirements  derived  on  the  basis  of  identified  needs  are  initially 
formulated  in  functional  terms,  rather  than  in  terms  of  specific  live  or 
simulated  inputs,  outputs,  or  other  implementing  solutions.  Immediate 
objectives  at  this  stage  are  to  determine  and  describe  exercise  requirements 
in  terms  of  the  following  parameters: 

(1)  Exercise  Configuration  —  the  particular  collection  of  personnel, 
operational  equipment,  and  operational  computer  programs  that  are  to  interact 
in  the  execution  of  a  given  exercise  mission. 

(2)  Evaluation  Exercise  Objectives  —  the  mission  element  or  collection 
of  mission  elements  to  be  exercised  by  a  given  configuration  of  the  operational 
system  for  purposes  of  evaluating  a  function  or  set  of  functions.  Special 
features  and  conditions  of  the  operating  environment  which  are  to  be 
incorporated  in  given  exercise  configurations  and  missions,  e.g.,  counter¬ 
measures  or  degraded  communications,  should  be  taken  into  account. 

(3)  Exercising  Conditions  —  conditions  affecting  conduct  of  exercises, 
e.g.,  off-station  vs.  on-station  considerations,  time-sharing  requirements, 
simultaneous  simulated  and  live  inputs. 

(U)  Special  Test  Data  Requirements  —  a  primary  objective  is  to  identify 
valid  and  useful  measures  of  performance,  considered  in  terms  of  (a)  the  total 
system,  (b)  intermediate  outputs  of  system  functions  or  subfunctions,  and 
(c)  special  components,  e.g.,  personnel.  In  general,  system/subsystem  output 
speeds  (or  latencies),  accuracies  (or  errors),  and  volumes  are  the  basic 
measures  to  be  defined  and  interrelated. 

In  addition  to  operational  command  evaluation  and  training  objectives  as  des¬ 
cribed  above  and  in  Block  D-l6,  requirements  for  Acquisition  Phase  development 
testing  should  also  be  considered  in  association  with  planning  for  exercise 
capability  implementation.  Hence,  additional  inputs  should  be  obtained  from 
the  test  planning  activities  initiated  in  Block  D-l^.  These  additional  require¬ 
ments  may  include  uses  of  the  exercise  capability  as  a  whole  to  satisfy  system 
test  objectives  during  Category  II  and/or  Implementation  Tests,  as  well  as  a 
variety  of  special  simulation  needs  associated  with  planned  Category  I  testing 
of  computer  program  CEIs. 

Applicable  Data  Item 

Q-117  Evaluation  Needs /Exercise  Requirements  Analysis. 
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D-18.  Define  Supporting  Computer  Program  Tasks 


The  activity  referred  to  in  this  Block  is  analogous,  technically,  to  that 
described  for  operational  CPCEIs,  under  the  preceding  Block  D-ll.  The  develop¬ 
ment  of  Part  I  Specifications  for  supporting  CPCEIs,  or  functions,  is  also  a 
process  of  proceeding  from  higher  to  progressively  lower  levels  of  detail • 

However,  as  indicated  earlier  in  Block  D- 9*  the  analysis  and  definition  of 
supporting  elements  involves  a  variety  of  technical  sources  and  approaches,  with 
varying  degrees  of  dependence  upon  the  specific  system  operations.  The  analysis 
and  design  of  many  utility  and  maintenance-diagnostic  functions  may  proceed  on 
the  basis  of  relatively  gross  information  about  system  operations.  On  the 
other  hand,  the  process  of  detailing  such  functions  as  simulation,  data  record¬ 
ing,  and  data  reduction  must  maintain  a  constant  closed-loop  relationship  with 
other  analyses  being  performed  during  Phase  B.  Much  of  the  basic  initial 
expansion  of  these  support  functions,  in  fact,  occurs  most  directly  in  the 
operational  CPCEI,  exercise  capability,  and  test  planning  flows,  as  described 
in  Blocks  D-ll,  D-lU,  D-15,  D-19,  D-20,  and  D-21.  Hence,  the  types  of 
activities  described  in  those  blocks  constitute  the  types  of  technical 
analyses  by  which  many  support  functions  are  defined  at  intermediate  levels  for 
input  subsequently  to  the  Block  D-22  activity,  for  detailed  integration  into 
Part  I  CPCEI  Specification  form. 
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D-19 .  Expand  Information  Processing  Requirements 


The  purpose  of  this  activity  is  to  further  expand  and  verify  the  information 
processing  functions  which  have  been  identified  as  logical  and  mathematical 
tasks  to  be  performed  by  the  operational  computer  program.  This  activity 
is  a  continuation  and  extension  of  the  activities  described  in  Blocks  D-ll 
and  D-15,  and  is  closely  coordinated  with  the  data  base  studies  described 
in  Block  D-20. 

Each  operational  computer  program  task  is  further  analyzed  to  identify  the 
data  input  sources,  the  data  received  from  each  data  input  source,  the 
information  processing  required,  feedback  responses,  the  data  output  destina¬ 
tions,  and  the  data  sent  to  each  data  output  destination.  The  method  of 
transmission,  frequency  of  transmission,  and  volume  of  data  are  determined 
for  the  data  received  from  each  data  input  source  and  the  data  sent  to  each 
data  output  destination.  Environmental  conditions,  critical  load  conditions, 
equipment  limitations,  and  major  sources  of  error  are  determined  for  each 
operational  computer  program  function  and  subfunction  and  evaluated  with 
respect  to  efficiency,  completeness,  design  feasibility,  interfaces,  and 
conformance  with  the  System  Specification.  In  addition,  for  each  task,  the 
applicable  system  limits  and  capacities  are  examined  and  evaluated  for  their 
effect  on  operating  efficiency  of  the  computer  program  as  a  whole. 

Alternative  logical  transformations  are  identified  and  evaluated  and  appropriate 
logical  transformations  are  selected.  Operational  equations  are  determined 
or  developed  and  evaluated  for  each  mathematical  computation,  giving  consid¬ 
eration  to  the  nature  and  accuracy  of  the  input  data,  the  accuracy  of  data 
from  which  parameters  for  the  equations  are  determined,  the  nature  and 
required  accuracies  of  output  data,  computer  program  operating  time,  computer 
program  space  requirements,  and  accuracy  requirements  for  the  operational 
equations . 
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D-20.  Expand  Data  Base  Requirements 


The  determination  of  data  base  requirements  is  a  continuing  activity  during 
Phase  B.  It  is  normally  based  on  gross  requirements  established  initially 
in  the  Conceptual  Phase  and  reflected  in  the  System  Specification  and  other 
prior  system  engineering  documentation.  While  certain  aspects  of  the  effort 
must  be  closely  coordinated  with  the  development  of  other  elements  of  the  Part 
I  CPCEI  Specification,  it  may  in  some  cases  constitute  a  sizeable  and  relatively 
independent  effort.  Often,  it  may  require  extensive  field  research  or  active 
participation  by  other  contractors  or  government  agencies. 

Depending  upon  the  system,  the  size  and  nature  of  the  data  base  may  vary  widely. 
Where  the  major  operational  function  of  the  system  centers  on  updating  and 
manipulation  of  a  very  large  data  base,  the  specification  of  data  base  elements 
may  constitute  a  correspondingly  large  portion  of  the  effort  involved  in 
developing  a  Part  I  CPCEI  Specification.  In  other  systems,  e.g.  ,  where  emphasis 
is  on  real-time  manipulation  of  continuous  inputs  from  system  sensors ,  the 
role  of  a  fixed  data  base  may  be  relatively  minor.  Also,  although  "data" 
constitute  the  basic  elements  which  are  manipulated  by  any  information  system, 
they  may  assume  a  variety  of  roles  in,  and  in  relation  to,  the  computer  program 
instructions.  The  rules  for  defining  the  data  at  the  Part  I  Specification 
level  will  vary  correspondingly.  Differences  in  both  the  derivations  and 
expressions  of  requirements  will  occur,  for  example,  depending  upon  such 
distinctions  as:  whether  the  data  are  to  be  input  continuously,  with  variable 
values,  during  the  course  of  computer  operation;  whether  the  data  are  input 
prior  to  computer  program  operation,  and  cannot  be  changed  in  the  course  of 
operation;  whether  the  data  are  constants  or  variables;  whether  the  data  exist 
in  source  input  form,  or  result  from  mathematical  or  logical  manipulation  by 
elements  of  the  computer  program;  etc. 

In  the  uniform  specifications  for  CPCEIs,  "adaptation  data"  represent  a  special 
segment  of  the  data  base  in  multi-site  systems.  The  distinction  is  made  for 
configuration  management,  rather  than  technical,  reasons.  These  are  items 
which  have  fixed  values  at  any  one  site,  but  which  vary  in  value  among  sites. 

The  function  of  this  distinction  is  to  avoid  a  potential  requirement  for  the 
computer  program  at  each  site  to  be  a  separate  CPCEI,  when  the  copies  are 
identical  except  for  those  selected  elements  of  fixed  data. 

In  preparing  the  Part  I  Specification,  all  types  of  data  should  be  identified, 
and  labels  and  definitions  provided  for  individual  data  items.  Eventually 
(see  Block  A-8,  A-22),  it  will  be  necessary  to  supply  coded,  actual  numerical 
values  for  all  items  which  are  quantitative  in  nature,  and  the  coded  states 
(e.g.,  wet,  dry,  in,  out,  etc.)  which  may  be  assumed  by  each  qualitative  item. 

At  this  stage,  the  level  of  definition  required  will  vary  for  different  classes 
and  subclasses  of  items.  In  general,  objectives  are  to  identify  the  item 
parameters  (i.e.,  units  of  measure,  range  of  possible  values,  and  precision  or 
accuracy  requirements),  rather  than  actual  numbers.  However,  the  actual 
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numerical  values  are  required  for  some  items,  e.g.,  computer  program  constants 
these  are  derived  in  the  normal  course  of  activities  described  in  Blocks  D-ll, 
D-15,  and  D-19.  For  some  classes  of  data,  it  may  be  necessary  to  specify 
the  source  and/or  methods  required  to  convert  source  data  values  into  forms 
suitable  for  use  by  the  computer  program. 
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D-21.  Define  Exercise  Capability  Design  Requirements 


Studies  conducted  during  the  Conceptual  Phase  identified  the  basic  requirements 
for  an  exercising  capability  to  be  used  for  training  and  evaluation.  These 
requirements,  and  gross  estimates  of  the  size  and  complexity  of  the  exercising 
capability  were  contained  in  the  System  Specification,  the  PTDP  and  other 
Conceptual  Phase  documentation.  The  analyses  described  in  Blocks  D-l6  and  D-17 
above  expanded  upon  the  Conceptual  Phase  studies  and  identified  in  detail  the 
training  and  evaluation  needs  and  associated  functional  requirements  for  an 
exercising  capability. 

The  activity  here  is  directed  toward  further  expansion  of  functional  require¬ 
ments  within  established  system  constraints  and  the  conduct  of  trade-off 
studies  and  analyses  to  arrive  at  detailed  design  requirements  for  the 
exercising  capability.  This  effort  involves  the  consolidation  of  all  identified 
functional  requirements,  as  well  as  identification  and  assessment  of  feasible 
methods,  techniques,  and  vehicles  for  implementation. 

Elements  to  be  analyzed  and  expanded  under  this  activity  may  include  the 
following: 

(1)  Detailed  performance/design  requirements  for  communications, 
facilities,  and  equipment  imposed  by  the  exercising  functions.  These  will 
include  both  impact  on  system  operational  equipment — e.g. ,  additional  computer 
capacity,  tape  units,  special  features  of  operating  consoles  —  and  detailed 
characteristics  of  special  operating  stations,  communications,  and  equipment 
such  as  special  data  input  devices,  problem  generators,  and  other  gear 
required  for  simulation,  monitoring/recording,  or  data  reduction.  Recommended 
performance  tolerances,  design  features,  constraints,  locations,  etc.  are 
input  to  Block  D-27  end  appropriately  reflected  in  interface  sections  of  the 
Part  I  Specifications  for  computer  programs. 

(2)  Identification  and  definition  of  personnel  functions  associated  with 

the  exercising  capability.  This  will  include  overall  management  of  the  canability 
end,  for  specific  exercises,  may  include  the  functions  of  exercise  planning 
and  preparation  of  materials  as  well  as  real-time  simulation  functions  to  be 
performed  during  conduct  of  exercises,  e.g,,  input  messages  from  external 
agencies.  This  information  provides  inputs  to  the  QQPRI  activity  (Block  D-25) 
for  further  analysis  and  definition  of  personnel  requirements. 

(3)  Detailed  performance/design  requirements  for  computer  programs. 

These  requirements  typically  encompass  the  functions  of  simulation,  operator 
self-instruction,  recording,  and  data  reduction.  They  are  input  to,  and 
coordinated  on  a  continuing  basis  with ,  the  preparation  of  Part  I  CPCEI 
Specifications.  Depending  upon  the  structuring  of  CPCEIs  for  a  given  system, 
the  simulation,  operator  self-instruction,  recording,  and  data  reduction 
functions  may  be  incorporated  in  operational  computer  programs  or  special 
support  computer  program  CEIs. 
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(4)  Problem  production  requirements.  Requirements  to  be  identified  and 
defined  in  this  area  are  special  facilities,  equipment,  computer  programs,  and 
procedures  for  generating  synthetic  input  "problems" ,  e.g.,  simulated  radar 
input  data,  synthetic  messages  or  other  inputs  from  specially-prepared  tapes, 
film,  scripts,  etc.  While  some  of  these  may  be  provided  by  deliverable 
support  computer  programs  for  problem-generation,  others  may  require  a 
centralized  facility,  particularly  for  exercises  of  system-wide  scope,  capable 
of  producing  problem  inputs  and  associated  aids  and  materials  as  required 
periodically  by  the  user  (see  Block  A-6). 

(5)  Procedural  data.  Manuals  and  guides  to  accompany  delivery  of  the 
system  to  the  user  may  include  program  users  manuals,  procedures  manuals  for 
planning,  preparing,  and  conducting  exercises,  positional,  guides  for 
simulation  personnel,  and  exercising  information  for  operating  personnel. 

The  proposed  implementation  resulting  from  analyses  and  design  trade  studies 
conducted  in  the  course  of  this  activity  will  normally  represent  a  combination, 
with  some  alterations,  of  the  comprehensive  requirements  previously  established 
by  analyses  of  training  and  evaluation  needs.  A  description  of  the  actual 
resulting  exercise  configurations,  missions,  conditions,  and  test  data, 
reflecting  losses  or  changes  in  desired  capabilities,  is  prepared  as  a  part 
of  this  activity  for  inclusion  in  the  Exercise  Capability  Implementation  Plan 
(Block  D-31). 

Refinement  of  performance/design  requirements  and  coordination  with  related 
activities  in  human  engineering  (Blocks  D-12  to  D-29),  personnel  requirements 
(Blocks  D-13  to  D-30),  operational  CPCEI  specification  (Blocks  D-ll  to  D-26), 
supporting  computer  program  specificaton  (Blocks  D-9  to  D-26),  and  test 
planning  (Blocks  D-l4  to  D-32)  are  continuing  activities  throughout  Phase  B. 

Applicable  Data  Item 

S-54-6.1  System/Design  Trade  Study  Reports. 
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D-22 .  Develop  Performance /Design  Requirements  for  Supporting  CPCEIs 


With  respect  to  general  methodology,  this  activity  for  supporting  CPCEIs 
corresponds  with  the  combined  activities  described  in  Blocks  D-19,  D-20, 

D-23,  and  D-24  for  analysis  and  definition  of  requirements  for  operational 
CPCEIs.  As  indicated  earlier  (Blocks  D-9,  D-l8),  timing  and  technical 
sources  of  information  used  in  determining  performance /design  requirements 
may  vary  markedly  for  the  different  classes  of  supporting  CPCEI  functions. 

By  this  time,  the  allocation  of  supporting  functions  to  CPCEIs  will  have  been 
determined.  For  certain  utility  and  maintenance-diagnostic  functions,  these 
determinations  may  have  been  established  in  the  System  Specification.  In  some 
cases,  they  may  have  been  allocated  to  another  system  segment,  associated  with 
the  system  computing  equipment.  Depending  primarily  on  administrative /manage¬ 
ment  considerations  (see  Block  D-26) ,  the  simulation,  recording,  and  data 
reduction  functions  may  be  grouped  in  various  ways  as  either  separate  CPCEIs 
or  combined  with  operational  functions.  In  any  event,  the  allocations  by  CPCEI 
establish  the  existence  and  levels  of  inter-CEI  functional  interfaces  to  be 
defined  in  each  CPCEI  Part  I  Specification;  the  relevant  interfaces  to  be 
detailed,  for  each  CPCEI,  are  those  with  each  other  interfacing  equipment  and 
computer  program  CEI. 

The  definition  of  supporting  computer  program  performance/design  requirements 
may  also  vary  in  completeness  and  level  of  detail  depending  on  the  type  of 
function.  For  example,  a  compiler  and  other  utility  tools  which  will  be 
required  for  developing  other  computer  programs  during  the  early  part  of 
Acquisition  should  be  complete,  and  some  degree  of  computer  program  design, 
coding,  or  testing  for  these  items  may  already  have  been  accomplished  by  this 
time.  For  items  which  depend  upon  design  of  the  operational  data  base  or  for 
simulation  or  data  recording/reduction  functions,  on  the  other  hand,  completion 
of  detailed  requirements  may  continue  to  lag  development  of  the  operational 
CPCEI . 

In  addition  to  the  development  of  Part  I  CPCEI  Specifications,  these  activities 
must  be  associated  with  the  formulation  of  detailed  technical  inputs  to 
various  other  system  engineering  and  planning  activities.  Inputs  will  include 
requirements  for  computer  program  user  manuals  (Block  D-33),  recommended 
equipment  and  facility  requirements  (Block  D-27),  expansion  of  the  System 
Specification  (Block  D-28),  and  personnel  requirements  (Block  D-25).  A 
Section  4,  "Quality  Assurance  Requirements",  must  be  developed  for  each  Part  I 
Specification,  and  a  Category  I  Test  Plan  prepared  for  each  supporting  CPCEI 
(Block  D-32). 
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D-23.  Detail  Operational  CPCEI  Performance/Design  Requirements 


The  purpose  of  this  activity  is  to  compile  the  detailed  performance  and  design 
requirements  of  the  operational  computer  program  CEI.  It  is  based  upon  inputs 
derived  from  the  activities  described  in  Blocks  D-ll,  D-15,  and  D-19,  and  must 
be  closely  coordinated  with  the  activities  of  Blocks  D-20  and  D-2^;  its  outputs, 
when  combined  with  the  outputs  of  D-20  and  D-2U,  will  constitute  Section  3, 
"Requirements11 ,  of  the  Part  I  CPCEI  Detail  Specification. 

Requirements  are  formulated  in  both  general,  descriptive  language  and  in 
precise,  quantitative  terms .  The  system  limits  and  capacities,  in  terms  of 
frequencies,  volumes,  time  limits,  etc.  of  data  to  be  handled  by  the  computer 
program  are  derived  principally  from  the  System  Specification.  These 
constitute  the  continuing  criteria  for  development  and  evaluation  of  the 
aggregate  of  functional  elements  which  constitute  the  CPCEI  at  the  Part  I 
Specification  level.  A  part  of  this  activity  is  to  develop  a  description  of 
the  total  CPCEI  in  terms  of  its  functional  elements,  illustrating  the  functions 
and  their  interrelationships.  Each  functional  element  is  detailed  with 
respect  to  source  and  type  of  inputs ,  destination  and  types  of  outputs ,  and 
associated  information  processing  subfunctions ,  specifying  pertinent  logical 
rules  and  conditions,  mathematical  equations  and  constants,  required 
accuracies /tolerances,  options,  modes  of  operation,  etc.  Human  performance 
requirements,  derived  from  the  System  Specification  and  from  the  efforts 
described  in  the  preceding  Blocks  D-12  and  D-15,  are  specified  in  terms  of 
required  display/control  characteristics  and  compatibilities,  timing,  and 
special  processing  to  facilitate  information-handling  and  decision-making 
functions  of  human  operators;  these  should  consist  in  large  part  of  statements 
of  human  engineering  design  criteria  which  are  reflected  in  the  detailed 
performance  requirements  specified  for  the  various  CPCEI  functional  elements. 

This  activity  also  includes  the  formulation  of  requirements  and  constraints 
which  will  apply  to  the  subsequent  Acquisition  Phase  computer  program  design, 
derived  largely  from  requirements  contained  in  the  System  Specification. 

These  may  include  requirements  for  the  computer  programming  language  to  be 
employed,  applicable  computer  programming  standards,  maximum  storage  utiliza¬ 
tion  for  specified  functions,  requirements  pertaining  to  segmenting  the  CPCEI 
into  computer  program  components  (CPCs),  testing  features,  and  design  for 
ease  of  malfunction  diagnosis  or  ease  of  design  modification. 
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D-24.  Detail  Functional  Interfaces 


The  purpose  of  this  activity  is  to  specify  in  detail  the  functional  interfaces 
between  the  operational  computer  program  CEI  and  other  computer  program  and 
equipment  CEIs.  In  large  part,  it  represents  an  additional  detailing  of  gross 
interface  information  contained  in  the  System  Specification,  based  upon 
Phase  B  analyses  performed  in  connection  with  the  activities  described  in 
preceding  Blocks  D-8,  D-ll,  D-12,  D-15,  D-19,  and  D-21. 

Functional  interfaces  with  each  other  computer  program  CEI  in  the  system  are 
identified  in  terms  of  specific  input  and  output  data,  data  rates,  message 
formats,  message  contents,  data  units  and  accuracies,  and  operational  limits. 

In  the  course  of  this  process,  it  is  important  to  insure  that  the  counterparts 
of  this  information  are  accurately  and  consistently  formulated  for  each  of 
the  other  interfacing  CPCEIs. 

Interfaces  between  the  computer  program  CEI  and  the  computer  and  associated 
equipment  are  defined  in  terms  of  all  such  relevant  characteristics  as 
memory  size,  word  size,  access  and  operation  times,  interrupt  capabilities, 
inputs  and  outputs,  buffers, and  special  capabilities.  Depending  on  the  degree 
to  which  characteristics  of  the  computer,  consoles,  and  other  associated 
equipment  are  known  at  this  time,  the  interfaces  may  or  may  not  be  defined  in 
complete  detail.  Some  of  the  interface  requirements  may  depend  upon  recommenda¬ 
tions,  or  recommended  alternatives,  for  subsequent  selection  and  detailing 
during  Phase  C  or  early  Acquisition  (see  Block  A-3).  In  the  case  of  consoles, 
in  particular,  recommended  performance/design  requirements  for  display  features 
and  manual  input  provisions  should  normally  stem  from  the  various  Phase  B 
activities  which  are  associated  with  the  development  of  Part  I  Specifications 
for  CPCEIs  (e.g..  Blocks  D-12,  D-15,  D-21,  D-22).  These  recommendations  are 
incorporated  in  the  materials  described  in  Block  D-27,  and  are  appropriately 
reflected  in  paragraph  3.2.1  of  each  Part  I  CPCEI  affected. 

Once  the  detailed  performance/design  requirements  for  relevant  console 
characteristics  have  been  derived  and  verified,  a  special  set  of  recommendations 
is  prepared  by  the  operational  CPCEI  design  activity  to  specify  the  numbers 
and  types  of  display  categories,  numbers  and  types  of  intervention  devices 
(e.g.,  buttons,  switches,  light  pen),  together  with  specific  wiring  connections 
to  be  provided  between  (l)  the  display  categories  and  intervention  devices  and 
(2)  field  or  bit  locations  in  computer  storage.  In  air  defense  systems,  the 
documents  containing  this  class  of  information  have  been  known  as  Variable-Display 
Equipment  (VDE)  Specifications.  For  systems  in  which  the  consoles  and  computer 
are  undergoing  a  parallel  development,  the  level  of  detail  required  to  complete 
this  information  may  not  be  available  prior  to  early  Acquisition.  It  must 
normally  be  provided  and  approved,  however,  prior  to  the  accomplishment  of 
detail  computer  program  design. 
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D-25.  Expand  Analysis  of  Personnel  Requirements 


This  block  is  an  extension  and  continuation  of  the  analyses  initiated  in  Block 
D-13.  It  is  intended  primarily  to  indicate  the  necessity  and  importance  of 
including  those  personnel  required  by  the  system  exercising  capability  and 
other  supporting  functions  (see  Block  D-21)  in  the  Personnel  Requirements 
Analysis.  It  also  emphasizes  the  requirement  for  continuing  the  personnel 
analysis  to  incorporate  the  more  detailed  information  relative  to  duty  positions 
and  personnel  qualifications  which  emerge  as  system  definition  becomes  more  precise. 

The  personnel  requirements  analyses  for  system  exercising  personnel  will 
follow  the  same  pattern  as  described  in  Block  D-13  for  operational  personnel. 

Special  attention  should  be  directed  to  the  following  areas: 

(1)  Requirements  which  the  system  exercising  capability  may  place  on 
operational  duty  positions  in  terms  of  new  tasks,  changed  workload,  special 
skills,  etc. 

(2)  The  availability  of  operational  personnel  freed  from  their  normal 
duty  positions  during  system  exercises  which  may  be  qualified  to  man  duty 
positions  required  by  the  system  exercising  capability. 

It  should  be  noted  that,  although  it  is  not  specifically  shown  on  the  flow 
diagram  in  Figure  3,  man-machine  task  analyses  must  frequently  be  performed 
with  reference  to  the  system  exercising  capability  for  the  same  reasons  they 
were  required  for  the  operational  system,  i.e.,  to  develop  and  verify 
performance/design  requirements  for  interfacing  computer  programs  and 
equipment,  and  to  help  establish  and  define  duty  positions  and  associated 
personnel  qualifications. 
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D-26.  Issue  Part  I  Detail  Specifications )  for  CPCEl(s) 

This  activity  consists  of  preparing  and  issuing  the  computer  program  CEI  Detail 
Specifications ) ,  Part  I,  in  approved  format.  The  contents  of  each  specifica¬ 
tion  were  generated  by  the  activities  which  were  previously  described  under 
Blocks  D-20,  D-22,  D-23,  and  D-24. 

A  separate  specification  must  be  prepared  for  each  computer  program  CEI.  At 
this  time,  all  CPCEIs  for  the  system  should  be  firmly  identified,  either 
confirming  or  revising  the  tentative  identifications  which  had  been  made  at 
earlier  points  during  Phase  A.  The  grouping(s)  of  operational  and  supporting 
computer  program  functions  into  one  or  more  CPCEIs  is  based  upon  a  variety 
of  both  technical  and  administrative  considerations.  In  general,  the  number 
of  separate  CPCEIs  proposed  by  a  single  contractor  should  be  held  to  a  minimum, 
in  the  interests  of  reducing  administrative  overhead.  However,  functions  may 
be  separated  into  two  or  more  CPCEIs  on  the  basis  of  such  factors  as  the 
following: 

(1)  When  the  functions  are  to  be  performed  at  separate  locations  or  times, 
and  for  separate  missions. 

(2)  When  the  schedules  for  development  and  qualification  are  significantly 
different. 

(3)  When  certain  computer  program  functions  have  potential  use  in  more  than 
one  system. 

(4)  In  all  cases,  when  configuration  management  during  the  Operational 
Phase  will  be  the  responsibility  of  different  agencies. 

The  computer  program  CEI  Detail  Specification,  Part  I,  specifies  the  per¬ 
formance,  design,  and  qualification  requirements  to  be  met  by  the  computer 
program  CEI.  As  a  product  of  the  Definition  Phase,  it  is  written  in  operational, 
logical,  and  mathematical  language.  Its  primary  function  during  Acquisition 
is  to  provide  the  detailed  statement  of  formal  requirements  against  which  the 
computer  program  design,  coding,  checkout,  test,  and  qualification  activities 
proceed  in  the  course  of  developing  the  CPCEI  and  its  Part  II  Detail  Specifica¬ 
tion.  In  addition,  the  Part  I  Specification  constitutes  (l)  the  basis 
for  approval  by  the  procuring  and  using  agencies  of  the  detailed  performance 
and  design  requirements  of  the  computer  progtam  CEI,  (2)  the  instrument 
which  defines  all  interfaces  between  the  computer  program  CEI  and  other 
computer  program  and  equipment  CEIs,  (3)  the  basis  for  the  development 
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of  the  support  documentation  associated  with  the  operation  and  use  of  the 
computer  program  CEI,  (U)  the  basic  vehicle  for  configuration  control  of  the 
computer  program  CEI  through  the  Acquisition  and  Operational  Phases  of  the 
system  development  process,  and  (5)  the  basis  for  the  preparation  of  the 
Category  I  Test  Plan  for  the  computer  program  CEI. 

Section  3,  Requirements” ,  constitutes  the  body  of  the  specification.  It 
contains  (l)  the  basic  specification  of  system  limits  and  capacities,  (2) 
the  definition  of  all  operating  functions  to  be  performed,  including  sequencing 
and  other  important  relationships,  (3)  the  sources  and  types  of  inputs,  (U) 
types  and  destinations  of  outputs,  (5)  detailed  requirements  for  logical  and 
mathematical  processing  functions,  (6)  detailed  definition  of  data  base 
elements,  (7)  human  engineering  requirements  incorporated,  (8)  definition  of 
functional  interfaces  with  equipment  and  other  CPCEIs,  and  (9)  design  require¬ 
ments  and  constraints  relating  to  standards,  organization,  design  for 
testability,  growth  potential,  etc. 

At  the  Part  I  Specification  level,  a  complex  CPCEI  will  normally  be  structured 
in  terms  of  a  set  of  major  functional  elements,  e.g.,  sequencing  control, 
data  inputs,  displays,  etc.  In  this  case,  the  specification  may  be  issued  as 
a  series  of  volumes,  rather  than  as  a  single  bound  document,  to  facilitate 
its  preparation  and  use. 

Section  U,  "Quality  Assurance  Provisions”,  establishes  requirements  for  develop¬ 
mental  testing  and  qualification  of  the  CPCEI  during  the  course  of  Category 
I  and  Category  II  test  programs.  It  identifies  all  (l)  requirements  associated 
with  test  support  of  the  contractor's  design  and  development  of  the  CPECI , 
e.g.,  for  use  of  government-furnished  equipment,  and  (2)  methods  by  which  the 
detailed  performance/design  requirements  contained  in  Section  3  will  be 
qualified  during  Preliminary  and  Formal  Qualification  or  Category  II  testing. 

As  issued  at  the  end  of  Phase  B,  Part  I  Specifications  should  be  complete  in 
sufficient  detail  to  provide  a  basis  for  evaluating  performance  adequacy  of 
each  proposed  CPCEI,  and  for  supporting  firm  estimates  of  Acquisition  Phase 
schedules  and  costs.  Often,  however,  a  substantial  amount  of  additional 
detail  may  be  required  in  order  to  complete  the  specification  at  the  level 
required  for  subsequent  computer  program  design,  coding,  and  qualification. 

In  general,  the  necessary  further  refinement  should  be  completed  prior  to  the 
initiation  of  detail  computer  program  design,  following  PDR  (see  Block  A-3). 

Applicable  Data  Item 

ESD  236  Contract  End  Item  Detail  Specification  (Computer  Program), 

Part  I 
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D-27 .  Compile  Recommendations  for  Equipment,  Facilities,  and  Communications 


The  purpose  of  this  activity  is  to  compile  recommended  design  requirements  for 
equipment,  facilities,  and  communications  based  upon  detailed  definition  of  infor¬ 
mation  processing  functions  and  tasks.  Inputs  to  this  activity  sure  derived  from 
man-machine  task  analysis  (Block  D-12),  display/control  requirements  (Block  D-15)« 
operational  computer  program  performance/design  requirements  (Block  D-23), 

Variable  Display  Equipment  (VDE)  specifications  (Block  D-2U),  exercise 
capability  design  requirements  (Block  D-2l),  test  planning  (Block  D-l4),  and 
supporting  computer  program  performance /design  requirements  (Block  D-22). 

At  this  stage  of  definition,  recommended  design  requirements  for  equipment, 

facilities,  and  communications  should  meet  one  or  more  of  the  following 

criteria:  (l)  necessity  for  accomplishment  of  defined  information  processing  tasks, 

(2)  advantageous  trade-off  with  computer  program  design  and/or  operation,  and 

(3)  potential  cost  savings  or  increased  effectiveness  in  meeting  predicted 
system  flexibility  requirements.  If  the  compilation  consists  of  recommenda¬ 
tions  for  modifications  or  additions  to  a  previously  defined  hardware  system, 
recommendations  must  be  referenced  to  the  known  value  parameters  of  timing, 
capacity,  format,  reliability,  configuration,  transfer  rates,  available 
peripheral  equipment,  and  structural  and  power  requirements.  If  the  hardware 
system  is  being  defined  in  parallel,  recommendations  should  reflect  known  or 
expected  requirements  for  hardware  configuration  and  operation,  taking  into 
account  interfacing  equipments  set  forth  at  gross  levels  in  the  System 
Specification. 

The  form  of  the  compilation  will  vary  from  system  to  system,  depending  on  the 
degree  and  kind  of  previous  definition  of  the  hardware  system.  In  general, 
however,  recommendations  should  be  organized  under  the  following  headings: 

( 1 )  Equipment 

(a)  Central  computer  storage  and  processing 

(b)  Input/output — sources,  destinations,  processing 

( c )  Peripheral  storage 

(d)  Operator  display /response  consoles — configuration,  signal 
assignments 

(e)  Special  purpose  test  equipment 

(f)  Utilization  of  system  hardware  for  test  purposes 
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(2)  Facilities 

(&)  Space  utilization 
(b)  Configuration 

(3)  Communications 

(a)  Automatic — inter-  and  intra- facility 

(b)  Manual —  inter-  and  intra-facility 

Comparative  analyses  of  the  recommendations  will  be  necessary  to  eliminate 
conflicting  or  redundant  recommendations.  Completeness  of  the  recommendations 
will  be  verified  through  analysis  of  the  functioned,  interface  requirements 
in  the  Part  I  Specifications  (Block  D-26)  and  review  of  the  Category  I  Test 
Plan  (Block  D-32),  Exercise  Capability  Implementation  Plan  (Block  D-31),  and 
Expanded  System  Specification  (Block  D-28). 
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D-28.  Complete  Expanded  System  Specification 

The  development  of  data  to  expand  those  portions  of  the  System  Specification 
which  a:e  allocated  to  system  segments  (see  Block  D-2)  is  a  continuing  activity 
throughout  Phase  B.  At  this  point,  inputs  resulting  from  analysis /design 
efforts  described  under  the  preceding  Blocks  D-8  through  D-25  are  compiled  and 
organized  into  the  System  Specification  format  for  inclusion  with  the 
contractor's  final  report. 

The  expanded  data  will  emphasize  detailed  definitions  of  functional  interfaces 
with  other  segments,  and  will  include  a  firm  list  of  the  contractor's  pro¬ 
posed  CPCEIs  (see  Exhibit  1,  AFSCM  375-1). 

It  is  a  normal  expectation  that  the  expansion  accomplished  at  this  time  and 
throughout  Phase  B  represents  additional  detail  to  clarify  and  amplify  basic 
requirements  contained  in  the  System  Specification  as  issued  and  baselined 
at  the  outset  of  the  Phase  B  effort.  Thus,  the  expansion  should  be  distinguished 
from  changes  to  the  basic  specification.  However,  basic  changes  may  also 
frequently  be  indicated  by  Phase  B  studies.  In  these  cases,  system  requirements 
ECPs/SCNs  may  be  prepaired  in  accordance  with  Exhibits  VII  and  VIII  of  AFSCM 
375-1  and  submitted  at  any  point  during  Phase  B;  participating  government 
agencies  and  contractors  are  informed  of  all  resulting  CCB  actions. 

Applicable  Data  Item 

C- 1-35. 1-1  System  Performance /Design  Requirements  General  Specification. 
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D-29 .  Compile  Operating  Procedures  and  Task  Analysis  Data 


This  represents  an  activity  to  prepare  products  of  Phase  B  human  engineering 
studies  for  inclusion  in  the  "System  Engineering  Documentation"  portion  of 
the  final  report.  The  scope,  content,  and  format  of  information  to  be 
reported  are  determined  selectively  for  each  system  and  specified  as  applicable 
in  the  CDRL  (DD  Form  lU23). 

Normally,  emphasis  is  placed  on  data  which  are  required  by  the  SPO  for  technical 
evaluation  of  the  contractor's  human  engineering  design  solutions.  This  may 
include  selected  analyses  and  trade-off  studies  generated  during  man-machine 
functions  allocation  (Block  D-8)  and  subsequently  (e.g..  Blocks  D-12  and  D-15), 
as  well  as  procedural  data  organized  in  a  form  to  provide  the  basis  for  sub¬ 
sequent  Acquisition  Phase  development  of  manuals  and  other  job  or  training 
aids. 

Procedural  data  should  normally  be  prepared  for  all  types  of  man-machine 
operating  stations.  In  general,  this  information  should  include  listings  of 
procedures  for  each  mode  of  normal  system  operation,  as  well  as  special  pro¬ 
cedures  for  anticipated  alternate  and  emergency  conditions,  identifying  avail¬ 
able  displays,  switch  action  sequences,  and  communications  with  other  internal 
or  external  operating  stations.  Supplementary  annexes  may  include  summaries 
of  all  display  formats,  switch  and  other  manual  inputs,  available  inter¬ 
operator  communicating  links,  and  identification  of  special  job  aids  required 
for  efficient  operation.  Information  in  this  area  may  also  provide  a  necessary 
adjunct  to  the  Part  I  CPCEI  Specification  as  a  basis  for  concurrence  by  the 
Using  Command. 

Detailed  task  analyses,  time-line  analyses,  and  trade-off  studies  are  normally 
conducted  and  reported  on  a  selective  basis,  emphasizing  those  operations 
which  are  critical  to  system  performance  or  for  which  significant  alternatives 
exist  with  reference  to  cost,  personnel  requirements,  or  development  time. 

Applicable  Data  Items 

Q-118  Human  Operator  Task  Analysis  for  Information  Systems. 

This  item  calls  for  data  which  are  generated  initially  as  aids 
to  equipment/computer  program  design  (Block  D-12).  With  minor  changes,  it 
can  also  be  used  to  cover  the  reporting  of  operating  procedures  as  end  products. 

S-5U-6.I  System/Design  Trade  Study  Reports. 

Applicable  reports  for  this  area  may  be  associated  with  man- 
machine  functions  allocation  accomplished  at  various  levels  during  activities 
described  under  Blocks  D-8,  D-12,  and  D-15.  As  indicated  in  the  Form  9,  the 
reporting  level  may  vary  considerably  as  a  function  of  each  study  objective. 


TO 


S-56-6.1  Time-Line  Sheets 


This  item  may  be  used  for  analysis  of  time-critical  man- 
machine  functions,  as  a  supplement  to  task  analysis. 
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D-30.  Complete  QQPRI  Report 


The  QQPRI  Report  is  based  primarily  upon  the  data  resulting  from  the  continuing 
Personnel  Requirements  Analysis  described  in  Blocks  D-13  and  D-25.  At  this 
time  the  report  is  preliminary  in  nature  and  will  be  updated  during  the 
Acquisition  Phase  (see  Block  A-l6). 

The  purpose  of  the  QQPRI  Report  is  to  provide  an  early  estimate  of  personnel 
required  to  operate,  maintain,  and  control  the  system.  Although  the  report 
must  normally  conform  to  rules  for  format  and  content  set  forth  in  MIL-D-26239A, 
the  principal  technical  content  will  be  contained  in  two  sections  as  discussed 
below: 

(1)  Position  Descriptions.  The  position  descriptions  contained  in 
Section  U  of  the  QQPRI  Report  represent  the  basic  source  of  information  on 
the  types  of  personnel  required  to  operate,  maintain, and  control  the  system. 

It  should  summarize  the  scope  of  responsibilities  and  the  nature  of  the  work 
performed  at  each  position  and  should  include,  as  a  minimum,  the  following: 

(a)  Identification  of  the  Air  Force  Specialty  (AFS)  required  and 
justification  of  any  recommendations  for  new  or  modified  AFSs. 

(b)  A  table  for  each  position  listing  the  job  operations  involved, 
and  for  each  job  operation  the  duties  and  tasks  required.  New  duties  and 
tasks  should  be  emphasized  along  with  any  conditions  likely  to  cause  errors 
or  which  impose  unique  personnel  or  training  requirements. 

(c)  A  listing  of  the  equipment  used,  with  emphasis  on  new  equipment 
and  associated  special  skills. 

(d)  An  estimate  of  the  time  required  to  accomplish  each  duty  and 
task,  the  location  where  each  is  performed, and  an  estimate  of  the  required 
frequency  of  performance. 

(e)  An  evaluation  of  each  task  for  probable  error  factors,  special 
handling  requirements,  difficult  control  manipulations,  special  skills  for 
interpreting  displays,  and  necessary  safety  precautions. 

(2)  Preliminary  Manning  Estimates  contained  in  Section  5  of  the  QQPRI 
Report  provide  the  basis  for  personnel  planning  agencies  to  consider  man¬ 
power  sources  and  problems,  and  for  the  using  command  to  develop  the  Unit 
Manning  Document  for  the  system.  This  section  should  state  basic  concepts 
and  assumptions  and  include  the  following: 

(a)  Manning  estimates  showing  the  number  of  personnel  required  to  per¬ 
form  the  duties  at  each  type  of  position  per  standard  working  shift  under  typical 
working  conditions.  Required  personnel  should  be  identified  by  AFS  and  AFSC  and 
an  estimate  provided  of  the  number  of  shifts  required  for  each  2k  hour  period. 
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(b)  Appropriate  diagrams  to  illustrate  the  overall  functional 
organization  required,  including  the  composition  of  major  organizational  units, 
crews,  or  teams. 

Applicable  Data  Item 

Q-103  Qualitative  and  Quantitative  Personnel  Requirements  Information, 

Part  I:  Field  and  Organization  Maintenance. 

Since  only  one  QQPRI  Report  is  prepared  for  a  system  as  a  whole,  the 
product  for  this  system  segment  will  represent  only  a  partial  input  to  the 
complete  report.  Information  to  be  supplied  under  the  computer  program 
system  segment  should  normally  cover  personnel  in  the  categories  of  (l)  system 
operators,  (2)  simulation/exercising,  and  (3)  computer  programming  support, 
including  contractor  technical  personnel  support  when  required.  Specific 
reporting  requirements  are  normally  determined  by  the  procuring  agency. 
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D-31.  Prepare  Exercise  Capability  Implementation  Plan 


The  Exercise  Capability  Implementation  Plan  describes  the  total  exercising 
capability  being  built  for  the  system,  defines  the  manner  in  which  the 
various  components  may  be  used  individually  and  collectively  in  various  con¬ 
figurations  to  provide  various  levels  of  system  exercising,  and  provides 
preliminary  planning  information  for  the  implementation  of  the  capability 
during  system  test  and  activation. 

Description  of  the  Exercising  Capability 

The  design  requirements  for  the  exercise  capability  initially  identified 
and  described  in  Block  D-21  above  and  continually  updated  and/or  modified  as 
a  result  of  system  engineering  activities  during  Definition  Phase  B  serves 
as  the  basis  for  describing  the  actual  exercise  capability  to  be  provided. 
This  description  should  identify  the  various  system  exercising  elements 
(e.g.,  equipment,  computer  programs,  aids,  exercising  personnel,  etc.), 
indicate  the  manner  in  which  the  elements  may  interact,  and  relate  the  total 
exercising  capability  to  the  training/evaluation  needs  identified  in  Blocks 
D-l6  and  D-17  above.  The  description  may  be  supported  by  block  diagrams, 
or  equivalents,  which  illustrate  relationships  among  elements  of  the 
exercising  capability  and  with  elements  of  the  operating  system. 


Definition  of  Levels  of  System  Exercising 

The  description  of  the  exercising  capability  provides  the  basis  for 
defining  how  each  of  the  elements  may  be  used.  Various  levels  of  system 
exercising  may  be  achieved  by  using  different  elements  or  groups  of  elements. 
Feasible  levels  should  be  identified,  and  for  each  level,  the  following  infor¬ 
mation  should  be  provided: 

(1)  System  element(s)  exercised. 

(2)  System  personnel  exercised. 

(3)  Exercising  capability  element(s)  required. 

(U)  Exercising  personnel  required. 

(5)  Training/evaluation  need(s)  satisfied. 

Any  training/evaluation  need  identified  in  Blocks  D-l6  and  D-17  which  is  not 
satisfied  by  one  or  more  exercising  levels  should  be  identified. 
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Implementation  Planning  Information 

Preliminary  planning  information  should  be  provided  which  indicates  the 
sequence  with  which  the  various  levels  of  system  exercising  will  be  required 
during  system  test  and  activation.  Early  requirements  for  system  exercising, 
such  as  may  typically  exist  for  Category  II  system  tests,  will  have  significant 
impact  on  the  development  and  test  schedule  for  the  exercising  capability 
itself  and  must  be  relatively  well  defined  at  this  point.  Requirements  state¬ 
ments  should  contain  the  following  information: 

(1)  Exercising  level(s)  required. 

(2)  Time  and  place  where  required. 

(3)  General  description  of  supporting  problem  materials,  procedures » 
and  aids  required. 

Applicable  Data  Item 

Q-120  Exercise  Capability  Implementation  Plan. 
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D-32. 


Prepare  Category  I  Test  Plan(s)  and  Inputs  to  Category  II  Test  Plan 


During  Definition  Phase  B,  the  rudimentary  planning  information  contained 
in  the  System  Test  Plan  should  be  expanded  by  the  contractor  into  a  separate 
Category  I  Test  Plan,  inputs  to  the  Category  II  Test  Plan,  and  for  a  multi¬ 
site  system,  inputs  to  the  Implementation  Test  Plan.  Where  multiple  con¬ 
tractors  are  involved,  each  contractor  prepares  plans  only  for  those  items 
contained  within  the  system  segment  for  which  he  has  responsibility.  In 
these  cases,  particular  attention  must  be  directed  towards  planning  for  the 
testing  of  interfaces  and  the  identification  of  testing  which  requires  the 
Category  II  test  environment. 

The  amount  of  detail  contained  in  the  test  plans  prepared  at  this  time  must 
necessarily  be  limited  to  a  level  consistent  with  the  state  of  development  of 
the  system  as  a  whole,  but  should  be  sufficient  to  allow  the  contractor  to 
develop  realistic  cost  estimates  for  purposes  of  preparing  a  firm  proposed 
for  the  acquisition  effort.  As  system  design  evolves  during  the  Acquisition 
Phase  and  more  detailed  design  information  becomes  available,  the  initial 
plans  will  be  expanded  (see  Block  A-9). 


Category  I  Test  Plan 

The  Category  i  Test  Plan  for  computer  programs  should  provide  the  overall 
planning  for  the  tests  necessary  to  confirm,  in  accordance  with  Section  h  of 
the  Part  I  CPCEI  Specification,  that  the  CPCEI  fulfills  the  requirements 
of  Section  3  of  the  Part  I  CPCEI  Specification.  Planning  information  should 
be  provided  for  the  following: 

(1)  The  location  of  the  tests,  and  schedules  relative  to  established 
milestones  in  the  overall  acquisition  process  for  the  system. 

(2)  General  methods,  requirements,  and  responsibilities  for  the 
preparation  of  input  data. 

(3)  General  procedures  for  test  conduct,  including  responsibilities 
for  test  direction,  operation,  and  observation. 

(U)  Requirements  for  computer  programs  (other  than  the  CPCEI  under 
test),  and  equipment  such  as  computers,  input/output  devices,  etc. 

(5)  Requirements  for  personnel,  including  statements  of 
responsibilities,  authority,  special  skill  or  knowledge,  etc. 
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(6)  Data  recording  and  reduction  requirements. 

(7)  Test  analysis  and  reporting  requirements. 

The  test  plan  should  be  organized  into  sections  corresponding  to  the  types 
of  Category  I  testing  of  computer  programs  as  follows: 

(1)  Computer  Programming  Test  and  Evaluation.  This  type  of  testing 
is  conducted  by  the  contractor  as  an  integral  part  of  the  design  and  develop¬ 
ment  process.  Information  concerning  this  type  of  testing  will  be  included 

in  the  test  plan  only  when  (a)  the  test  is  intended  to  be  the  only  source  of 
data  for  qualifying  specified  functions,  or  (b)  the  test  requires  the  use  of 
government  facilities  or  other  support  items  such  as  computing  and  peripheral 
equipment . 

(2)  Preliminary  Qualification  Tests  (PQT).  PQTs  are  formal  tests 
oriented  towards  verifying  during  the  process  of  detail  design  and  developed 
prior  to  the  availability  of  the  complete  CPCEI,  that  selected  individual 
functions  meet  specified  requirements.  The  test  plan  should  indicate  those 
functions  and  portions  of  functions  which  will  probably  not  be  examined  during 
PQT,  and  indicate  the  reasons  for  their  exclusion.  PQTs  serve  the  primary 
purpose  of  demonstrating  the  contractor’s  scheduled  progress  towards  meeting 
the  computer  program  design  objectives.  Although  preliminary  in  nature,  PQT 
test  results  may  be  utilized  by  the  monitoring  agency  to  verify  detailed  per¬ 
formance  characteristics  of  individual  computer  program  components  (CPCs) 
which  may  not  be  feasible  to  examine  during  subsequent  formal  qualification 

of  the  completed  CPCEI. 

PQTs  will  normally  be  conducted  at  a  contractor's  development  facility,  typically 
using  controlled  inputs  specifically  prepared  for  test/demonstration  purposes. 

The  test  plan  should  outline  the  probable  sequence  of  individual  and/or 
assembled  CPC  tests,  identify  special  simulation,  recording,  equipment,  or 
other  support  requirements  for  the  PQT  program.  In  addition  to  identifying  all 
test  tools  which  will  be  required,  the  test  plan  should  indicate  how  and  to 
what  degree  these  test  tools  will  have  been  validated  for  this  purpose. 

(3)  Formal  Qualification  Tests  (FQT).  FQTs  normally  bear  the 
principal  burden  of  qualifying  the  CPCEI.  They  consist  of  formal  tests/ 
demonstrations  using  the  completely  assembled  CPCEI,  oriented  towards  verifying 
the  full  spectrum  of  functional  requirements  set  forth  in  the  Part  I  Detail 
Specification.  In  most  cases,  FQT  will  rely  heavily  upon  simulated  and 
controlled  inputs  which  can  be  designed  to  cover  the  expected  ranges  of 
system  operational  modes  and  conditions. 
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Complex  operational  CPCEIs  may  be  scheduled  for  formal  qualification 
at  the  Category  II  test  site,  as  the  probable  first  location  at  which  the  system 
will  be  available  in  its  full  operational  configuration.  In  these  cases, 
scheduling  of  the  FQT  will  normally  be  (a)  prior  to  the  start  of  Category  II 
testing  per  se,  in  order  to  qualify  the  CPCEI  for  the  Category  II  tests,  but 
(b)  following  a  period  of  contractor  adaptation,  installation,  and  checkout 
of  the  CPCEI  in  the  previously-installed  and  tested  operational  equipment/ 
facilities. 

For  less  complex  CPCEIs,  or  for  those  which  are  relatively  insensitive  to  the 
system  operation,  formal  qualification  may  be  accomplished  at  the  contractor's 
development  facility.  This  is  especially  likely  to  be  the  case  for  a  utility 
computer  program.  These  items  may  often  be  qualified  through  use  in  testing 
and  operation  of  other  CPCEIs,  rather  than  by  special  tests. 

Formal  qualification  of  a  support  CPCEI  (e.g.,  simulation  or  data  reduction) 
may  represent  a  special  case.  In  general,  the  qualification  of  functions 
which  are  sensitive  to  the  proper  performance  of  the  operational  computer 
program,  including  maintenance/diagnostic  functions,  may  often  be  delayed 
until  late  in  the  Category  II  testing  period. 

Inputs  to  the  Category  II  Test  Plan 

The  Category  II  Test  Plan  to  be  prepared  by  the  SPO  during  Definition 
Phase  C  or  early  in  the  Acquisition  Phase  will  be  based  upon  inputs  provided 
by  each  contractor  during  Definition  Phase  B.  The  computer  program  contractor's 
inputs  have  special  significance,  since  computer  programs  are  the  elements 
which  most  directly  implement  the  operational  functions  of  an  information 
processing  system. 

The  inputs  prepared  at  this  time  should  emphasize  those  aspects  of  system 
performance  which  involve  complex  interactions  among  computer  programs, 
personnel,  equipment,  communications ,  and  facilities  during  live  operations 
of  the  total  system  and  should  accomplish  the  following: 

(1)  Identify  all  test  objectives  which  must  be  fulfilled  in  order 
to  satisfy  the  requirements  of  the  System  Specification  and  any  CPCEI  Part  I 
Specification  which  may  contain  requirements  which  could  not  be  satisfied 
during  Category  I  testing. 

(2)  Outline  in  general  terms  the  test  methods  and  procedures  for 
satisfying  the  test  objectives  identified  above. 

(3)  Describe  the  types  of  data  which  should  be  collected,  and  how 
they  should  be  reduced,  analyzed,  and  reported.  Special  requirements  for  data 
recording  and  reduction  computer  programs  and/or  equipment  should  be  identified. 
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(h)  For  systems  which  require  implementation  testing  at  follow-on 
sites,  identify  specific  measures  of  system  performance  (barometers  and  figures 
of  merit)  to  be  verified  during  the  Category  II  test  program  for  the  purpose 
of  providing  basic  criteria  for  proving  the  acceptability  of  subsequent  sites. 

(5)  Identify  the  numbers  and  types  of  trained  personnel  required. 
Where  system  level  training  over  and  above  the  individual  training  provided 
by  ATC  is  a  requirement,  preliminary  plans  should  be  developed  for  its 
accomplishment.  This  should  include  identification  of  special  requirements 
for  facilities,  equipment,  training  materials,  etc. 

(6)  Identify  simulation  requirements.  Although  Category  II  testing 
will  use  live  inputs  to  the  maximum  extent  possible,  it  is  frequently  not 
technically  or  economically  feasible  to  rely  entirely  on  such  inputs.  For 
air  defense  systems,  for  example,  it  is  not  feasible  to  provide  a  sufficient 
number  of  appropriate  aircraft  to  test  the  system  at  or  near  maximum  load 
levels.  In  such  cases  it  may  be  desirable  to  utilize  the  system  exercising 
capability  to  provide  exercise  problems  for  use  during  Category  II  tests. 

Such  a  requirement  should  be  identified  at  this  point  in  time  since  it  would 
have  an  impact  on  the  scheduled  development  of  the  exercise  capability  and  the 
production  of  exercise  problem  materials  (see  Blocks  D-26  and  D-27). 

(7)  Provide  Personnel  Subsystem  planning  information  relative  to 
the  evaluation  of: 

(a)  The  adequacy  of  the  man/machine  interface  (human 

engineering). 

(b)  The  adequacy  of  the  environmental  conditions  in  terms 
of  permitting  effective  personnel  performance. 

(c)  The  adequacy  of  technical  data. 

(d)  The  adequacy  with  which  system  malfunctions  can  be 
detected,  isolated,  analyzed  and  corrected  by  trained  military  personnel 
using  only  authorized  equipment  and  technical  data. 

(e)  The  adequacy  of  safety  precautions  in  all  operations 
and  maintenance  activities. 

(f)  The  adequacy  of  communications  between  and  within 

functions. 
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Applicable  Data  Items 

T-103  Category  I  Test  Plan/Procedures  (Computer  Programs) 

This  Form  9  combines  the  Category  I  Test  Plan  and  Category  I 
Test  Procedures  into  one  form.  Only  the  first  part  of  the  form  dealing  with 
planning  information  will  be  completed  at  this  time.  The  portion  dealing 
with  procedures  will  be  completed  during  the  Acquisition  Phase. 

T-106  Category  II  Test  Plan/Procedures 

The  appropriate  paragraphs  of  this  Form  9  will  be  utilized  by 
the  contractor  to  provide  planning  information  relative  to  the  system  segment 
for  which  he  has  development  responsibility.  Integration  of  the  inputs  from 
the  various  contractors  into  the  Category  II  Test  Plan  for  the  entire  system 
is  a  SPO  responsibility  ,  which  may  be  supported  by  a  GSE/TDC  or  system 
contractor(s). 
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D-33.  Provide  Inputs  to  the  Definition  Phase  B  Final  Report 


The  Contractor's  Phase  B  Final  Report  is  one  of  the  four  major  products  of  the 
Definition  Phase  (along  with  the  PSPP,  the  PCP  and  an  updated  MCP).  Typically 
it  will  contain  the  type  of  information  indicated  below,  conforming  to  the 
four-part  outline  set  forth  in  AFSCM  375-^  (Part  I,  Ch.  3). 

Part  I  Introduction  and  Brief  Summary 

Part  II  Technical  Report 

(1)  Trade  Study  Conclusions 

(a)  Man-machine  function  allocations 

(b)  Development  of  mathematical  equations 

(c)  Alternate  information  processing  flows 

(d)  Display  design 

(e)  Manual  input  alternatives 

(2)  System  Engineering  Documentation 
(&)  Functional  allocations 

1.  Computer  program  functions 

2.  Operator  functions 

(b)  Operator  task  analysis 

(c)  Training  and  Evaluation  needs  analyses 

(d)  System  Exercise  Capability  implementation  plan 

(e)  Inbuilt  simulation  and  test  capabilities 

(f)  Data  reduction  capabilities 

(g)  Computer  program  development  tools  required  (Utility) 

(h)  Personnel  requirements  information  for  operational, 
computer  program  support,  and  simulation/exercising 
personnel. 
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(i)  Recommended  design  requirements  for  equipment, 
communications,  and  facilities. 

(3)  The  System  Specification 

(a)  Definition  and  list  of  computer  program  CEIs. 

(b)  Functional  allocations 

(h)  Part  I  Computer  Program  CEI  Detail  Specification s ) 

(a)  Operational  Requirements 

(b)  Support  requirements 

(c)  Utility  requirements 

(5)  Inventory  Equipment  Requirement  Detail  Specifications 
(Not  applicable) 

(6)  Contractor  Data  Requirements  List 

Part  III  Contractor's  Acquisition  Phase  Program  Management  Plans 

(1)  Typical  planning  areas  to  be  covered  are  listed  below. 

(a)  Organization  and  Personnel  Management 

(b)  Technical  Management 

(c)  Detailed  Integration  During  the  Acquisition  Phase 

( d )  Development 

(e)  Test 

(f)  Installation  and  Checkout 

(g)  Financial 

(h)  Procurement 

(i)  Personnel  and  Training 

(2)  Among  other  planning  activities  which  may  be  included  as 
parts  of,  or  together  with,  the  plans  selected  from  the 
typical  planning  areas  listed  above  are:  personnel 
subsystem,  government-furnished  property  (GFP),  PERT/Cost, 
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test  site  responsibilities,  configuration  management,  and 
exercise  capability  installation  and  orientation 

Part  IV  Contractor's  Firm  Proposal  for  the  Acquisition  Phase 

(1)  Statement  of  Work  for  the  Acquisition  Phase 

(2)  Cost  Data — PERT/Cost  Report 

(3)  Contract  Terms  and  Conditions 

( 4 )  Incentive  Features 

(5)  Operating  PMN 

(6)  Schedules 

As  indicated  by  the  above  outline,  the  major  system  engineering  inputs  to  the 
final  report  will  be  contained  in  Part  II,  Technical  Report.  In  practice,  the 
Technical  Report  will  consist  of  a  number  of  separate  documents  corresponding 
to  those  described  in  Blocks  D-26  through  D-32  above.  Additional  system 
engineering  inputs  which  might  be  expected  are  discussed  below. 

The  contents  of  Part  III,  Contractor's  Acquisition  Phase  Program  Management 
Plans ,  will  vary  widely  from  system  to  system  in  terms  of  the  number  and 
types  of  program  management  plans  required.  Planning  areas  contained  in  the 
outline  which  normally  require  system  engineering  inputs  and  would  typically 
be  of  concern  to  the  computer  program  contractor  include  the  following: 

(1)  Detailed  Integration  During  the  Acquisition  Phase 

( 2 )  Development 

(3)  Test 

(4)  Installation  and  Checkout 

(5)  Personnel  and  Training 

System  engineering  inputs  to  Part  IV,  Contractor's  Firm  Proposal  for  the 
Acquisition  Phase,  would  typically  be  centered  in  the  following  two  areas: 

(l)  Statement  of  Work  (SOW).  The  SOW  to  be  prepared  at  this  time 
represents  an  expansion  and  refinement  of  the  Specimen  Statement  of  Work 
contained  in  the  contractor's  proposal  for  Definition  Phase  B,  and  consists 
of  a  detailed  description  of  each  task  to  be  performed  by  the  contractor 
during  the  Acquisition  Phase.  The  basis  for  the  detailed  task  descriptions 
will  be  contained  in  the  system  engineering  documentation  completed  during 
Definition  Phase  B. 
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(2)  Schedules .  Schedule  information  contained  in  the  proposal  will 
necessarily  be  constrained  by  the  overall  development  schedule  for  the  system. 
Aspects  of  the  system  engineering  effort  during  Phase  B  which  would  be 
reflected  here  are  the  verification  that  the  system-level  schedules  are 
realistic  from  the  standpoint  of  the  computer  program  contractor,  and  the 
identification  of  milestones  in  the  computer  program  development  process  which 
must  coincide  with  specific  system-level  milestones.  Where  appropriate, 
lower  level  scheduling  information  should  be  provided  for  computer  programming 
events  which  fall  between  identified  system  milestones. 

Applicable  Data  Items 

S-17-12.0-1  Technical  Reports. 

S-5-l^.O  Proposal-Technical. 

(See  also  Blocks  D-l6,  D-17,  D-26,  D-28,  D-2$>,  D-30,  D-31,  and  D-32.) 
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PHASE  C  -  REVIEW  AND  DECISION 


D-31*.  Perform  Technical  Evaluation  and  System  Engineering  Synthesis 

The  SPO  technical  evaluation  accomplished  at  this  time  will  normally  emphasize 
the  proposed  Part  I  CPCEI  Specification,  the  updated  System  Specification, 
together  with  associated  system  engineering  documentation  contained  in 
Part  II  of  the  contractor  final  report,  and  the  technical  proposal  for 
Acquisition  Phase  development.  Objectives  are  to  determine  whether  Phase  B 
obligations  have  been  met,  through  analysis  of  the  technical  adequacy  and 
completeness  of  Phase  B  products,  and  to  determine  recommended  source(s) 
for  the  ensuing  Acquisition  Phase. 

When  major  deficiencies  are  in  evidence,  or  when  major  advantages  would  accrue, 
the  SPO  may  undertake  to  combine  features  proposed  by  competing  contractors 
into  an  improved  single  approach.  This  activity  constitutes  a  "system 
engineering  synthesis";  it  involves  certain  potential  limitations  with  respect 
to  proprietary  rights  and  subsequent  contracting,  and  may  require  extensive 
additional  system  engineering  effort  during  Phase  C. 
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CHAPTER  IV  ACQUISITION  PHASE 


A.  GENERAL 

The  Acquisition  Phase  is  initiated  by  issuance  of  the  System  Management  Direc¬ 
tive,  which  approves  the  design  requirements  baseline,  PSPP,  PCP,  and  MCP  (if 
applicable) , and  identifies  the  availability  of  funds  and  manpower  resources 
required  for  acquisition  of  the  system. 

Based  upon  the  typical  cost  and  schedule  requirements  for  system  equipment  and 
facilities,  the  Acquisition  Phase  may  consist  of  two  overlapping  parts,  each 
of  which  may  be  contracted  for  separately.  The  first  is  referred  to  as 
development,  and  encompasses  the  activities  of  design,  development,  and 
associated  developmental  testing.  The  second  part  is  a  production  or  construc¬ 
tion  effort,  which  may  include  the  construction  of  one-of-a-kind  hardware. 

It  is  to  be  noted  that  only  the  first  type  of  contract  applies  to  activities 
subsumed  under  the  computer  programming  system  segment,  due  to  the  relative 
insignificance  of  time  and  costs  involved  in  computer  program  duplication 
(production),  together  with  the  fact  that  effort  and  costs  to  build  the  first 
item  are  integral  with  development  of  the  Part  II  CPCEI  Specification. 

The  Part  I  Specifications  for  computer  programs,  like  those  for  equipment  and 
facilities,  constitute  the  baseline  requirements  against  which  design  and 
development  activities  proceed  during  Acquisition  towards  the  completion  of 
computer  program  CEIs  and  their  Part  II  Detail  Specifications. 

However,  the  Part  II  CPCEI  Specifications  may  typically  be  completed  later  in 
the  Acquisition  Phase  than  would  often  be  true  of  prime  equipment  items  in 
the  same  system,  both  because  of  typical  requirements  for  using  system 
equipment  to  qualify  the  computer  programs  and  because  of  the  absence  of 
required  lead-time  for  their  "production".  Specific  planning  considerations 
relative  to  equipment  and  other  system  events  are  amplified  in  the  narratives 
of  this  chapter. 
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B.  DIAGRAM  OF  THE  ACQUISITION  PROCESS 


Activities  shown  on  the  "Acquisition  Process"  diagram  in  Figure  U  are  represented 
at  four  horizontal  levels  to  correspond,  in  a  general  way,  with  the  major  sub¬ 
processes  of  (top  to  bottom):  (a)  computer  program  design  and  coding, 
including  reviews  and  inspections,  (b)  test  activities,  (c)  personnel  subsystem 
and  handbooks,  and  (d)  development  of  elements  required  for  the  system  exercise 
capability. 

The  development  process  is  not  shown  separately  on  the  diagram  for  different 
subclasses  of  operational,  support,  utility,  and  maintenance-diagnostic 
CPCEIs,  since  the  design/development /test  events  shown  apply  in  the  same 
general  sequence  to  all  CPCEIs.  For  simplicity,  the  sequential  phasing  of 
events  is  shown  which  would  be  typical  of  major  operational  CPCEIs,  and  the 
variations  in  relative  timing  which  would  normally  occur  for  certain  supporting 
items  are  noted  in  the  event  narratives. 

An  important  activity  not  represented  on  the  diagram  consists  of  a  necessary 
continuation  of  the  Definition  Phase  effort  which  led  to  the  Part  I  CPCEI 
Specifications.  This  effort  is  frequently  referred  to  as  "operational 
design",  in  distinction  with  the  Acquisition  Phase  computer  programming  design 
activities  which  lead  to  the  Part  II  Specifications.  Although  usually  at  a 
reduced  level  as  compared  with  the  Definition  Phase,  the  operational  design 
activity  must  normally  be  maintained  on  a  continuing  basis  throughout  the 
Acquisition  Phase  for  the  primary  purposes  of  (a)  supporting  the  computer 
program  design  in  areas  related  to  detailed  performance/design/test  require¬ 
ments  contained  in  the  approved  Part  I  CPCEI  Specifications,  and  (b)  developing 
changes  to  the  Part  I  Specifications  as  the  needs  for  such  changes  are 
identified  and  approved  during  Acquisition. 

In  the  diagram,  and  in  the  narratives,  Formal  Qualification  Test  is  represented 
as  occurring  at  the  Category  II  site,  following  the  installation  and  checkout 
of  system  equipment.  This  is  the  sequence  which  has  actually  been  followed 
in  recent  system  programs,  and  which  may  continue  to  be  necessary  in  some 
future  cases.  However,  it  should  be  emphasized  that  it  is  generally  desirable 
to  accomplish  formal  qualification  prior  to  this  time,  to  the  degree  that 
available  simulation,  equipment,  and  other  relevant  factors  Justify  determining 
that  the  tests  are  actually  valid  with  respect  to  established  performance 
requirements  for  a  given  CPCEI. 
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c. 


NARRATIVES 


A-l.  Avard  Contract(s) 


Contract  awards  are  based  on  the  evaluations ,  system  engineering  synthesis , 
and  negotiations  conducted  during  Phase  C,  and  incorporate  firm  constraints 
and  requirements  established  through  DoD/USAF  approvals.  Significant  SPO 
activities  preceding  contract  awards  include  preparation  of  the  SPP,  issuance 
of  real-estate  directive,  activation  of  inter-agency  agreements,  updating  of 
the  design  requirements  baseline,  and  appointment  of  Category  I  and  II  test 
directors. 

The  contract  awarded  at  this  time  for  the  computer  program  system  segment 
will  be  a  development  contract  covering  the  design,  development,  and  testing 
of  computer  programs,  together  with  the  development  of  design  requirements 
baseline  changes,  associated  personnel  subsystem,  procedural  data,  and 
system  exercising  products.  The  contract  will  normally  include  tasks  for 
computer  program  adaptation,  installation,  and  checkout  at  system  site 
locations,  and  necessary  support  of  Category  II  and  Implementation  Tests. 


A-2.  Implement  System  Engineering  Procedures 


This  activity  includes  expansion  and  revision  of  system  engineering  documenta¬ 
tion  and  source  data  which  had  been  established  and  maintained  during  the 
Definition  Phase  (see  Block  D-7),  to  reflect  firm  decisions  and  revisions 
resulting  from  higher  headquarters  review,  preparation  of  the  SPP,  and  contract 
negotiation.  At  this  time,  procedures  may  be  augmented,  using  automated  data 
handling  techniques  where  indicated,  to  assure  efficient  maintenance  of  the 
data  for  continuing  utilization  by  contractor  technical  personnel  during 
Acquisition. 

Where  there  is  a  large  number  of  computer  programming  and  other  technical 
personnel  involved  in  the  development  effort,  it  is  also  important  to  establish 
at  this  time  a  common  set  of  standards,  practices,  conventions,  and  techniques 
to  govern  the  process.  Some  degree  of  consistency  is  imposed  by  uniform 
computer  program  specifications  and  uniform  requirements  for  other  items  of 
contractor  data.  Beyond  these  standard  requirements  for  deliverable  documenta¬ 
tion,  however,  there  exist  essentially  no  detailed  standards  for  computer 
programming  which  are  comparable  to  the  wealth  of  Federal  and  military 
specifications  and  standards  which  govern  the  design  and  construction  of 
equipment.  Nevertheless,  each  contractor  can  define  and  disseminate  rules  of 
standard  practice  to  be  followed  internally  in  such  areas  as:  terminology, 
with  respect  both  to  the  given  system  and  to  permissible  technical  terms, 
abbreviations,  characters,  symbols,  etc.;  computer  programming  techniques; 
standard  coding  and  flow  diagram  conventions;  rules  for  identifying  data  and 
other  CPCEI  elements,  updating,  document  duplication  and  labeling,  etc. 
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A-3.  Update  Part  I  CPCEI  Detail  Specification s ) 


The  updating  of  Part  I  Specifications  which  occurs  at  this  time  should  reflect 
both  expansions  of  information  and  changes  necessitated  by  events  following  the 
completion  of  Phase  B.  Changes  may  be  dictated  by  the  System  Management 
Directive.  In  some  cases,  both  changes  and  expansions  may  be  based  upon  firm 
selection  and  approval  of  specific  hardware,  computer  programs,  or  communications 
CEIs  from  among  alternatives  which  had  been  proposed  by  competing  Phase  B 
contractors . 

Specifications  submitted  at  the  end  of  Phase  B  must  be  sufficiently  complete 
in  both  scope  and  detail  to  provide  an  adequate  basis  for  costs,  schedules, 
and  evaluation  of  performance.  However,  the  level  of  detail  required  for 
those  purposes  may  often  be  more  gross  than  the  level  which  the  Part  I 
Specification  for  a  computer  program  CEI  must  contain  in  order  to  provide  an 
adequate  basis  for  subsequent  computer  program  development  and  qualification. 

Most  of  the  necessary  additional  detail  should  have  been  generated  by  this 
time,  through  continued  contractor  effort  during  the  Phase  C  holding  period 
and/or  at  the  outset  of  Acquisition.  The  specification  should  be  fully 
complete  in  all  sections  prior  to  PDR. 

The  updated  specifications  are  submitted  to  the  SPO.  Following  approval,  they 
are  turned  over  to  the  configuration  management  division  for  formal  control. 

All  subsequent  changes  or  expansions  are  initiated  and  formally  processed  as 
changes  to  the  design  requirements  baseline  (see  Exhibits  IX  and  XXI,  ESD 
Exhibit  EST-l). 
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A-U.  Accomplish  Preliminary  CPCEI  Design 


The  purpose  of  this  activity  is  to  translate  the  Part  I  Specification  into 
a  design  approach  to  the  overall  CPCEI  preparatory  to  Preliminary  Design 
Review  (Block  A-7).  Based  upon  performance  requirements  and  design  constraints 
established  in  the  Part  I  Specification,  the  CPCEI  preliminary  design  phase 
is  concerned  with  establishing  the  design  framework  within  which  subsequent 
design, coding,  and  testing  of  computer  program  components  can  be  accomplished, 
in  the  process  of  developing  a  Part  II  Detail  Specification  for  the  CPCEI. 

Determine  Computer  Program  Functional  Areas 

Integral  with  preparation  for  preliminary  program  design  is  the  analysis 
which  must  be  made  of  the  functions  to  be  performed  by  the  CPCEI  in  order  to 
determine  the  computer  program  structure  which  would  permit  the  required 
functions  to  be  performed  most  efficiently.  This  activity  entails  a  determina¬ 
tion  of  computer  program  functional  areas  which  are  comprised  of  one  or  more 
subprograms,  each  of  which  performs  a  particular  task  related  to  the  given 
function.  Functional  areas  of  a  computer  program  have  a  close  correspondence 
to  the  functions  described  in  the  Part  I  Specifications.  For  example,  the 
computer  program  functional  area  of  tracking  executes  the  tracking  function  as 
described  in  the  Part  I  Specification.  However,  this  correspondence  may  not 
exist  in  every  instance.  Bookkeeping  tasks  may  be  treated  in  the  Part  I 
Specification  as  tasks  implicit  in  the  performance  of  other  functions; 
efficient  program  design  may  demand  that  bookkeeping  be  performed  by  a  group 
of  subprograms  comprising  a  functional  area.  In  performing  this  analysis, 
the  following  should  be  taken  into  consideration: 

(1)  Inputs  from  and  outputs  to  other  CPCEIs  according  to  types, 
quantities,  frequency,  variations  in  sources,  and  destinations. 

(2)  Information  processing  functions  explicit  in  the  CPCEI  Part  I 
Specifications  and  the  implicit  information  processing  functions  which  are 
necessary  to  facilitate  the  performance  of  the  explicit  functions. 

(3)  Operating  characteristics  of  the  computer  in  which  the  CPCEI  is  to 
operate . 

(k)  The  language(s)  in  which  the  CPCEI  is  to  be  coded. 

(5)  The  test  environment  available  for  accomplishing  Test  and  Evaluation 
and  other  test  environments. 

(6)  Explicit  and  implicit  requirements  for  ease  of  malfunction 
diagnosis  and  simplicity  of  modifying  the  CPCEI. 


Determine  Computer  Program  Component  Structure 

Computer  programs  are  designed  so  that  sets  of  code  are  grouped  in  such 
a  manner  that  each  grouping  serves  a  specific  purpose.  These  groupings,  termed 
computer  program  components  (CPCs),  may  consist  of  such  elements  as  subprograms 
which  serve  the  purpose  of  a  task  within  a  functional  area,  or  a  data  dictionary 
(COMPOOL)  which  serves  the  purpose  of  defining  storage  locations,  or  control 
tables  which  contain  data  used  by  more  than  one  subprogram  in  the  execution 
of  its  specific  task.  Structuring  the  computer  program  into  CPCs  facilitates 
testing  in  that  each  subprogram  can  be  tested  as  an  isolable  entity  through 
the  use  of  controlled  inputs  and  analysis  of  the  outputs;  CPCs  also  permit 
optimum  utilization  of  data  storage  and  operating  time,  to  the  degree  that 
the  transfer  of  CPCs  in  and  out  of  various  storage  elements  can  be  performed 
in  such  a  way  as  to  achieve  the  desired  balance  between  operating  time  and 
computer  program  size. 

Storage  requirements  and  hardware  limitations  may  also  influence  the  determina¬ 
tion  of  CPCs.  Space-sharing  of  functions  and  commonality  of  subfunctions  will 
also  be  influencing  factors. 

Many  of  the  inputs  to  and  outputs  from  CPCs  are  obtained  from  other  CPCs  or 
destined  for  other  CPCs  within  the  CPCEI.  These  are  intra-CPCEI  communication 
requirements,  or  CPC  interfaces.  Accomplishing  the  CPCEI  preliminary  design 
requires  choosing  a  method  or  methods  which  will  be  employed  among  the  CPCs 
to  achieve  a  definition  of  these  interfaces. 

Determine  Operating  Sequence  and  Timing 

Once  functional  areas  have  been  defined,  it  is  possible  to  establish  a 
sequence  of  their  operation.  Requirements  such  as  several  iterations  of  opera¬ 
tion  during  a  cycle  in  order  to  minimize  timing  of  internal  transfers  and 
to  most  fully  utilize  the  existing  hardware  capabilities  must  be  considered. 
Additional  factors  influencing  the  sequence  include  the  order  of  data  inputs, 
minimization  of  internal  transfers  of  data  for  each  CPC's  operating  environ¬ 
ment,  and  allocation  for  common  usage  of  data. 

Timing  of  the  operation  of  the  entire  CPCEI  must  also  be  defined.  The  timing 
information  may  be  of  two  types:  (l)  definition  of  conditions  which  will 
affect  the  overall  operating  time  of  the  CPCEI,  (2)  estimations  of  these 
lengths  of  time  for  some  CPCEIs.  For  certain  functions,  this  definition 
is  critical  to  obtain  an  adequate  design;  however,  this  may  not  be  necessary 
for  all  areas.  In  the  design  of  an  assembler,  for  example,  estimations  for 
the  length  of  operating  time  need  not  be  gathered.  However,  the  conditions 
which  affect  the  operating  time  of  the  assembler  should  be  defined. 
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Define  Executive  Control  Program  Requirements 

Since  the  control  functions  are  defined  in  the  Part  I  Specifications,  it 
is  part  of  the  design  activity  to  assign  these  functions  to  CPCs.  Some 
elements  of  the  control  function  include  system  start  up,  operation  of 
individual  CPCs,  automatic  error  detection  and  recovery,  control  of  system 
timing,  initiation  of  program  interrupts  caused  by  equipment  requirements  and 
multi-processing  considerations,  etc. 

It  is  essential  that  the  activities  necessary  to  produce  the  executive  control 
program  be  initiated  as  early  as  possible,  as  the  design  of  this  is  critical 
to  the  overall  interaction  of  the  computer  program  system.  Also,  the  avail¬ 
ability  of  the  control  functions  at  the  earliest  possible  date  will  greatly 
facilitate  the  testing  of  other  functions. 

Define  the  Data  Base 

All  information  contained  in  the  computer  and  its  auxiliary  storage,  which 
is  not  included  in  the  instruction  set,  is  considered  to  be  the  data  base. 

This  information  is  used  for  CPC  inter-communication,  for  data  manipulation, 
for  specific  environmental  parameters,  etc.  To  accomplish  the  definition  of 
the  data  base,  data  items  and  sources  must  be  identified  and  format,  scaling, 
and  expected  usage  detailed. 

The  data  base  is  structured  to  allow  efficient  utilization  of  the  available 
storage  space  by  the  CPCs.  In  determining  the  organization  of  the  data  base, 
considerations  are  given  to  timing  requirements,  frequency  of  use,  whether 
the  data  are  permanently  required  for  the  system  cycle  or  whether  only  temporary 
storage  is  used.  The  organization  of  the  data  may  be  partially  derived  from 
the  analysis  of  the  flow  of  information,  and  further  detailed  as  storage  for 
data  used  by  CPCs  is  identified.  The  data  base  will  require  further  refining 
as  additional  work  is  completed  and  more  detailed  information  is  available 
at  the  level  of  individual  CPCs. 

Allocate  Program  and  Data  Storage  Space 

Plans  for  the  CPCEI's  use  of  program  and  data  storage  space  such  as  core 
memory,  drums ,  and  discs  should  be  described  in  order  to  accomplish  the  pre¬ 
liminary  design.  For  some  CPCEIs  it  may  be  reasonable  to  layout  the  proposed 
use  for  each  register,  file,  channel,  etc.  In  most  instances,  however,  this 
detail  is  not  known  as  early  as  the  preliminary  design  stage  and  the  proposed 
use  is  described  in  more  general  terms.  Estimates  for  requirements  of 
quantities  of  data  and  program  storage  should  be  obtained.  This  may  include 
estimates  on  storage  requirement  lengths  for  each  CPC  in  the  CPCEI,  or  it  may 
be  estimates  of  lengths  of  groups  of  CPCs  which  are  functionally  or  logically 
related  in  their  space  allocation  requirements.  For  data  storage  requirements, 
estimates  may  be  made  for  the  large  quantities  of  data  divided  according  to 
logical  relationships  and  for  some  smaller  quantities  of  data  where  the 
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necessity  has  been  determined.  These  quantities  of  data  will  be  described 
according  to  their  purpose,  i.e.,  names  of  types  of  data  which  are  allocated, 
in  addition  to  their  estimated  lengths. 

The  description  of  the  use  of  storage  space  can  include  layout  charts.  In 
some  instances,  it  may  be  determined  that  it  is  more  desirable  for  the  CPCEI 
itself  to  allocate  the  available  space.  In  this  instance,  it  is  useful  to 
describe  the  algorithm  which  will  be  coded  into  the  CPCEI  to  allocate  the  space. 

Perform  Trade-Off  Studies 

During  prelimianry  CPCEI  design,  trade-off  studies  must  often  be  made  to 
determine  the  most  effective  CPCEI  design.  In  each  of  these  studies,  space, 
timing,  and  feasibility  are  critical  factors  to  be  considered.  Additionally, 
for  many  CPCEIs,  extensive  and  frequent  changes  may  be  expected  after  the 
initial  operational  implementation.  The  ability  to  implement  these  changes 
quickly  and  easily  is  also  an  essential  consideration  in  these  studies. 

When  trade-off  studies  are  required  to  choose  among  significant  alternative 
approaches,  they  should  be  documented  to  reflect  the  alternatives  considered 
and  the  basis  for  the  final  decision,  in  order  to  facilitate  review  and  con¬ 
sideration  of  related  design  decisions  at  subsequent  levels. 
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A-5.  Initiate  Refinement  of  Operator  Procedures 


This  activity  is  concerned  with  the  continued  analysis  and  refinement  of  the 
operator  procedures  developed  during  Definition  Phase  B  as  described  in  Block 
D-29. 


Initially,  the  results  of  system  engineering  synthesis  occurring  in  Definition 
Phase  C  (see  Block  D-3*0  should  be  examined  for  changes  that  might  be  required 
to  the  operator  procedures  previously  prepared.  Any  required  changes  should 
be  made  at  this  time  and  any  procedures  not  prepared  earlier  because  of 
uncertainties  regarding  interfaces  should  be  completed. 

Analyses  of  manual  and  man-machine  tasks  should  be  initiated  or  continued 
as  required  to  optimize  procedural  efficiency  and  allocation  of  functional 
elements  and  to  reflect  additional  detailed  information  resulting  from  the 
acquisition  process.  Close  coordination  must  be  maintained  with  design 
personnel  throughout  the  Acquisition  Phase  so  that  as  problems  arise  requiring 
modification  or  revision  to  the  Detail  Specifications,  human  engineering 
personnel  cam  participate  in  their  resolution  and  evaluate  any  resultant 
changes  for  impact  on  operator  positions  or  procedures. 

The  refined  task  analyses  and  procedures  data  provide  the  basis  for  the 
compilation  of  materials  for  handbooks  and  guides.  A  handbook  is  normally 
prepared  for  each  type  of  operator  position  in  the  system,  providing  necessary 
information  for  the  performance  of  that  position.  It  should  be  designed  as  a 
complete  reference  document  to  supplement  on-the-job-training  and  cross¬ 
training,  and  may  also  be  required  for  use  as  a  basic  text  for  initial  training 
of  the  operators.  In  addition  to  detailed  procedures,  the  information  to  be 
compiled  at  this  time  should  identify: 

(1)  Content  and  format  of  all  displays  available  or  forced  to  the  position, 
including  information  concerning  the  frequency  with  which  displayed  data 

are  updated,  and  including  limits,  capacities,  and  tolerances  for  each  element 
of  displayed  information. 

(2)  Switch  actions  available  to  the  position  and  a  description  of  what 
occurs  within  the  system  as  a  result  of  each  switch  action. 

(3)  Special  job  aids  available  at  the  position,  with  detailed  descriptions 
of  how  they  should  be  used,  under  what  circumstances,  and  for  what  purpose. 

(U)  Communications  links  available  to  other  internal  and  external 
positions,  including,  when  available,  general,  operating  standards  or  procedures 
to  be  followed  in  their  use. 

In  addition  to  positional  handbooks,  operator  training  guides  will  usually 
be  prepared  for  use  in  initial,  replacement,  and  cross-training  of  operators. 

In  contrast  to  the  handbooks ,  which  are  reference  compendiums  oriented  to 
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specific  operator  positions,  the  operator  training  guides  are  systematic 
expositions  of  the  way  in  which  system  data  are  received,  processed,  and 
output  in  the  performance  of  major  system  functions.  Information  derived 
from  operator  task  and  procedural  analyses  form  a  major  portion  of  the  guides. 

The  ongoing  task  analysis  and  refinement  of  operator  procedures  should  also 
provide  the  basis  for  revising  and  updating  the  preliminary  QQPRI  Report 
prepared  during  the  Definition  Phase  (see  Block  D-30).  Changes  required 
should  be  supported  with  appropriate  justification  from  task  analyses  or 
other  sources. 

This  activity  may  also  be  expected  to  provide  inputs  to  the  Category  I  and  II 
PSTE  planning  effort  (Blocks  A-9,  A-17,  A-24 ,  and  A-29).  These  inputs  would 
consist  largely  of  detailed  procedural  data,  and  the  identification  of  critical 
tasks  requiring  special  attention  during  the  PSTE  effort. 
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A-6.  Initiate  Development  of  Exercise  Problem  Generating  Capability 


The  activity  to  be  discussed  here  is  concerned  with  the  development  of  a 
centralized  capability  for  generating  problem  materials  for  full-scale  system 
exercises.  This  centralized  capability  may,  in  some  instances,  represent  the 
only  source  of  exercise  problem  materials  available  to  the  system.  In  other 
instances,  it  may  augment  a  problem  material  production  capability  internal 
to  the  system.  In  air  defense  systems,  for  example,  both  the  SAGE  and 
BUIC  systems  have  computer  programs  at  each  site  which  are  capable  of 
producing  materials  on  a  limited  basis,  e.g.,  for  exercising  individual 
sites,  or  a  restricted  number  of  adjacent  sites.  For  nation-wide 
air  defense  exercises,  however,  a  centralized  capability,  in  this  case  external 
to  either  system,  produces  the  required  exercise  materials. 

Although  the  decision  that  a  centralized  problem  generating  capability  will  be 
required  may  have  been  made  as  early  as  the  Conceptual  Phase,  actual  develop¬ 
ment  is  dependent  upon  having: 

(1)  The  system  exercising  capability  fully  defined. 

(2)  Implementation  schedules  for  system  exercising  established. 

(3)  Development  contracts  awarded. 

For  the  purpose  of  this  discussion  it  is  assumed  that  the  contractor  selected 
for  developing  the  system  exercising  capability  is  also  responsible  during  system 
acquisition  for  the  delivery  of  any  exercise  problems  which  require  a  central¬ 
ized  facility  for  their  production.  As  described  here,  the  centralized 
capability  is  not  a  deliverable  product  of  the  system  development  contract.  It 
should  be  noted,  however,  that  some  or  all  of  the  problem  generating  capabilities 
and  materials  may  be  in  the  form  of  CEIs  (or  parts  of  CEIs)  and  associated  data, 
handbooks,  etc.  which  would  be  specified,  controlled,  developed,  tested,  and 
delivered  with  other  parts  of  the  system. 

The  magnitude  of  the  effort  required  for  developing  the  centralized  capability 
may  vary  widely.  At  one  extreme,  only  hand-prepared  scripts  and  scenarios 
must  be  produced  and  no  special  equipments  or  computer  programs  required.  At 
the  other  extreme,  the  capability  must  produce  a  wide  variety  of  exercise 
materials  and  may  require  a  complex  combination  of  equipment,  computer  programs, 
and  supporting  procedures  and  aids.  The  centralized  capability  for  producing 
nation-wide  air  defense  problems  represents  an  example  of  the  latter  case. 

That  capability  utilizes  special  computer  programs,  approaching  the  operational 
computer  programs  in  complexity  and  number  of  instructions,  and  special  equip¬ 
ments  to:  (l)  produce  maps,  lists,  and  other  exercise  aids,  (2)  generate  radar 
target  data  film  which,  when  input  to  special  exercising  equipment,  exercises 
the  system’s  radars,  and  (3)  generates  problem  input  tapes  which  contain  all 
of  the  inputs  to  the  Direction  Center  necessary  to  simulate  a  war-time 
environment. 
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In  addition  to  initiating  the  development  of  equipment  and/or  computer  programs 
which  may  be  required  to  produce  exercise  material,  this  activity  should  also 
be  concerned  with  the  development  of  techniques,  procedures,  and  data  necessary 
to  effectively  utilize  the  capability.  Of  particular  importance  is  the 
identification  and  collection  of  the  data  which  must  be  input  to  the  capability 
to  produce  exercise  material.  To  be  effective,  the  exercise  material  should 
be  based  on  detailed  data  concerning  the  anticipated  operational  environment 
in  which  the  system  to  be  exercised  must  perform.  In  the  air  defense  system 
example,  these  data  include  such  diverse  items  as:  site  location,  inter¬ 
site  communications,  performance  characteristics  of  enemy  aircraft,  enemy  ECM 
capability,  and  the  number,  types,  performance  characteristics,  and  armament  of 
interceptor  aircraft  available  to  the  defense. 
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A-7. 


Conduct  Preliminary  Design  Review  (PDR) 


The  Preliminary  Design  Review  is  a  formal  technical  review  of  the  basic  design 
approach  for  the  CPCEI  (or  functionally  related  groups  of  CPCEIs)  resulting 
from  the  preliminary  design  activity  discussed  in  Block  A-U.  The  purpose 
of  the  PDR  is  to  establish  the  integrity  of  the  design  approach,  verify 
compatibility  with  the  Part  I  Detail  Specification,  and  verify  functional  inter¬ 
faces  with  other  CEIs  prior  to  the  start  of  detail  design  activity  (Block  A-10). 

The  PDR  for  CPCEIs  is  accomplished  by  reviewing  the  following: 

(1)  Storage  Allocation  Charts.  These  charts  describe  the  manner  in 
which  various  elements  of  the  computer  program  system  have  been  allocated 
to  available  computer  storage. 

(2)  Computer  Program  Functional  Flow.  This  is  a  flow  chart  which 
summarizes  the  functions  to  be  performed  by  individual  computer  programs 
and  shows  the  sequence  of  operation. 

(3)  Control  Functions  Description.  This  is  a  description  of  the 
executive  control  features  and  start/recovery  functions  for  the  computer 
program  system.  This  includes  the  method  used  to  initiate  system  operation, 
together  with  features  to  enable  recovery  from  system  malfunctions. 

(h)  Structure  and  Organization  of  the  Data  Base.  This  is  a  description 
of  the  structural  layout  and  allocation  of  storage  requirements  of  the  system 
data  base,  identifying  the  types  and  characteristics  of  data. 

(5)  Interface  Requirements.  Although  these  were  identified  in  the  Part 
I  CEI  Specification  and  approved,  it  is  important  that  they  be  reviewed  again 
at  this  time.  Interface  constraints  identified  would  include  such  things  as 
word  length,  message  formats,  available  storage  within  the  computer,  and 
timing  considerations. 

(6)  Trade-Off  Studies.  This  is  information  developed  from  studying 
alternate  methods  and  techniques  of  designing  computer  program  CEIs. 

Applicable  Data  Item 

ESD  289  Minutes  of  Formal  Reviews  and  Inspections 
References 

AFSCM  375-1,  Exhibit  XIV 

ESD  Exhibit  EST-3,  Chapter  3 
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A-8.  Initiate  Collection  and  Compilation  of  Data  Base 


This  activity  consists  of  the  collection  and  subsequent  compilation  of  those 
data  elements  of  the  computer  program  which  must  be  obtained  from  sources  other 
than  the  contractor  responsible  for  the  computer  program  development.  The 
data  base  is  initially  identified  in  Conceptual  Transition  phase  (Block  C-5) 
and  subsequently  expanded  in  the  Definition  Phase  (Block  D-20).  Data  elements 
to  be  collected  include  those  which  describe  the  natural  environment  of  the 
system,  characteristics  of  weapons  to  be  employed  in  the  system,  interfacing 
system  characteristics,  and  other  similar  data.  These  data  frequently  must  be 
acquired  from  several  agencies,  e.g. ,  contractors  and,  in  some  instances,  from 
other  military  services.  Hence,  much  of  the  actual  data  collection  may  be 
performed  by  government  agencies,  through  arrangements  accomplished  by  the  SPO. 
The  contractor  should  assist  by  specifying  schedules  to  assure  timely  avail¬ 
ability  for  such  dependent  events  as  computer  program  testing  and  adaptation. 

The  data  must  be  compiled,  organized,  and  verified  for  consistency,  completeness, 
and  conformance  with  requirements  of  the  Part  I  Specifications ) .  Some  of  the 
data,  e.g.,  adaptation  data  or  weapons  characteristics,  may  require  documenta¬ 
tion  in  English  language/decimal  form  for  subsequent  delivery  to  the  user.  Also, 
the  contractor  must  normally  document  the  processes  to  be  employed  for 
encoding  the  data  in  forms  required  for  insertion  into  the  computer  program 
(see  Block  A-22)  and  suitable  for  listing  in  the  Part  II  CPCEI  Specification s) . 

In  cases  where  the  data  base  elements  are  likely  to  undergo  frequent  or 
extensive  change  during  the  Operational  Phase,  a  supporting  computer  program 
which  will  perform  the  task  of  encoding  the  data  into  machine  language  of  the 
computer  may  be  included  in  the  requirements.  In  these  cases,  the  design  of 
the  supporting  computer  program  must  be  reviewed  for  compatibility  with  the 
composition  of  the  data  base  elements.  A  user  manual  will  normally  be  required 
for  this ,  as  well  as  for  other  supporting  computer  programs ,  for  availability 
at  AI&C  (Block  A-22). 


A-9.  Expand  Category  I  Test  Plan 


The  purpose  of  this  activity  is  to  expand  and  update  the  Category  I  Test  Plan(s) 
prepared  during  the  Definition  Phase  (see  Block  D-32).  The  primary  objective  of 
this  expansion  process  is  the  establishment  of  basic  Category  I  requirements, 
methods,  and  criteria  that  will  (l)  provide  the  monitoring  agency  with  detailed 
information  required  for  effective  planning  and  management  of  the  Category  I 
test  activity,  (2)  identify  test  requirements  that  are  to  be  satisfied  during 
Category  II  testing,  (3)  provide  the  basis  for  early  review  and  analysis  of  the 
contractor's  testing  program  by  the  monitoring  agency  and  (U)  establish  the 
basis  for  preparation  of  Category  I  Test  Procedures, 

The  Category  I  Test  Plan  developed  in  the  Definition  Phase  identifies,  based 
upon  the  quality  assurance  requirements  specified  in  Section  h  of  the  Part  I 
Specifications:  the  requirements  for  personnel,  equipments,  facilities,  and 

supporting  computer  programs;  the  quantitative  criteria  for  verification  of 
performance;  the  documentation  and  control  procedures  to  be  employed;  gross 
level  scheduling  information;  and  the  functions  and  functional  interactions  to 
be  tested.  Expansion  of  this  information  will  usually  result  in  the  prepara¬ 
tion  of  a  set  of  test  plans  that  define  in  greater  detail  the  Category  I 
test  requirements,  methods,  and  criteria.  Additional  detailed  information 
required  for  this  activity  results  primarily  from  updating  and  more  precise 
definition  of: 

(1)  Equipment  configuration,  capabilities,  and  schedules. 

(2)  Facility  and  communication  linkage  capabilities  and  schedules. 

(3)  Computer  program  design  and  associated  development  schedules. 

(U)  System  environmental  conditions. 

With  the  contract  awards  for  computer,  display,  test,  or  other  equipment 
items.  Part  I  Specifications  and  other  documents  become  available  to  provide 
firm  and  more  detailed  information  relating  to  equipment  design  requirements 
and  schedules.  Pertinent  technical  information  will  include  such  items  as 
addressing  logic,  interrupt  logic,  storage  capacities,  etc.,  which  affect  both 
Part  I  CPCEI  Specifications  and  subsequent  computer  program  design,  and  which 
must  also  be  reflected  in  the  updated  Category  I  test  planning  with  respect 
to  requirements,  methods,  and  criteria.  The  test  schedules,  locations,  and 
personnel  requirements  will  also  require  expansion,  resulting  from  known  equip¬ 
ment  availability  schedules.  Factors  to  be  taken  into  account  will  include  firm 
availability  dates  for  operationally  configured  equipment,  as  well  as  schedules 
and  characteristics  of  other  equipment  items  needed  to  support  the  computer 
program  development  and  testing  efforts. 

Decisions  made  in  the  Definition  Phase  as  to  methods  to  be  used  in  qualifying 
CPCEIs  should  be  re-evaluated  and  assessed  as  a  result  of  updating  of  the 


Part  I  Specifications,  preliminary  CPCEI  design  decisions,  and  the  equipment- 
related  information  discussed  above,  as  well  as  any  required  modifications  to 
the  original  test  plan.  For  example,  plans  to  use  test  and  evaluation  data 
for  qualification  may  be  affected  if  equipment  other  than  that  to  be  employed 
in  an  operational  environment  is  to  be  used  during  the  test  and  evaluation 
time  period.  Qualification  requirements  in  this  case  may  result  in  changing 
planned  PQT  and/or  FQT  requirements. 

Testing  will  usually  require  the  use  of  facilities  and  communication  linkages, 
some  of  which  may  be  newly  constructed  and/or  installed.  The  expanded  test 
plan  should  account  for  construction  and  installation  schedules,  and  specify 
requirements  and  schedules  for  their  use.  In  addition,  existing  capabilities 
should  be  examined  for  use  in  testing.  Requirements  for  temporary  capabilities 
should  be  specified  and  trade-offs  made  by  assessing  the  costs  involved  and 
the  effects  on  the  total  schedules  in  providing  temporary  capabilities,  in 
adhering  to  existing  facility  and  communication  linkage  schedules,  and  in 
accelerating  these  schedules.  The  assessment  should  consider  such  factors 
as  installation  schedules,  verification  schedules,  backup  capabilities, 
anticipated  downtimes,  and  scheduled  maintenance  requirements.  Expansion  of 
test  plan  requirements  covering  these  factors  should  be  specified  and  pro¬ 
visions  made  for  all  required  interfaces  and  time-phasing  of  requirements. 

Information  resulting  from  the  preliminary  CPCEI  design  activities  described 
in  Block  A-U  influences  the  expansion  of  the  Category  I  Test  Plan  primarily 
in  such  areas  as:  identification  of  discrete  subprograms,  CPCs,  and  functional 
interfaces  requiring  testing;  test  input  expectations;  test  output  requirements; 
test  tool  capabilities;  and  overall  computer  program  development  schedules, 
including  the  establishment  of  PDR  and  CDR  schedules  and  requirements;  manpower 
and  computer  time  requirements;  test  procedure  production  schedules;  and  time 
required  for  test  conduct  and  test  result  analysis. 

All  quality  assurance  requirements  specified  in  Section  U.1.3  of  the  updated 
Part  I  Specifications,  and  the  manner  in  which  they  are  to  be  satisfied,  are 
identified  in  the  expanded  test  plan.  Requirements  that  cannot  be  qualified 
during  Category  I  testing  are  specified.  Requirements  of  this  nature  are 
typically  those  that  require  interfacing  equipment,  personnel,  facilities, 
and/or  other  systems  that  are  unavailable  until  the  Category  II  testing  time 
period,  and  for  which  the  development  of  sophisticated,  high-cost  test  tools 
or  equipment  is  not  economically  or  technically  feasible.  Other  require¬ 
ments  qualified  during  the  Category  I  time  period  may  be  qualified  by  indirect 
means.  For  example,  certain  utility  functions  may  be  qualified  in  this  manner 
since  the  validity  of  the  utility  functions  is  established  through  their  use 
in  developing,  maintaining,  and  debugging  other  CPCEIs. 

Environmental  requirements,  the  details  of  which  are  not  known  sufficiently  to 
incorporate  them  in  the  test  planning  during  the  Definition  Phase,  will' 
be  specified  in  the  expanded  test  plan  as  they  become  available.  For  example, 
if  a  manned  interceptor  is  acquired  in  conjunction  with,  or  as  part  of  the 
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system,  characteristics  requiring  testing  and  the  methods  to  be  employed  should 
be  included  in  the  expanded  plan  as  the  Part  I  Specifications  for  the  aircraft 
become  available. 

The  expanded  plans  are  reviewed  by  the  monitoring  agency  for  accuracy  of 
technical  data,  appropriateness  of  schedules  and  interface  requirements,  and 
completeness  of  coverage.  The  computer  program  test  and  evaluation  activity 
(Block  A-lU)  may  be  initiated  in  some  instances  prior  to  the  completion  of  the 
updated  test  plans,  where  data  resulting  from  the  operation  of  a  sub¬ 
program  may  be  required  for  PDR.  The  test  and  evaluation  effort  and  the 
production  of  test  procedures  should,  however,  be  consistent  with,  and  be 
guided  by  the  requirements  set  forth  in,  the  expanded  test  plan. 

Applicable  Data  Item 

T-103  Category  I  Test  Plan/Procedures  (Computer  Programs) 

References 

(1)  ESD  Exhibit  EST-2,  Chapter  U,  Block  18 

(2)  AFSCM  37^-U,  Part  I,  Chapter  U,  Block  18 
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A- 10.  Accomplish  Computer  Program  Component  (CPC)  Logical  Design 


The  purpose  of  this  activity  is  to  expand  and  amplify  the  content  of  the  CPCEI 
preliminary  design,  and  to  produce  documentation  and  flow  diagrams  describing 
the  detailed  logic  to  be  used  in  developing  code  for  individual  CPCs  (see 
Block  A-13).  The  input  to  this  phase  is  the  CPCEI  preliminary  design, 
including  the  results  of  any  analysis  studies  which  are  performed  during  this 
phase  that  are  pertinent  to  the  implementation  of  the  CPCEI.  Examples  of 
the  latter  would  be- studies  concerned  with  problems  of  transferability  from  an 
intermediate  to  the  operational  computer  ,  and  studies  concerned  with  problems 
of  operating  times  and  their  relationship  to  the  efficiency  of  the  CPCEI. 

The  CPC  logical  design  phase  overlaps  both  the  CPC  preliminary  design  phase 
(Block  A-4)  and  the  CPC  coding  phase  (Block  A-13).  Inputs  concerning  technical 
feasibility  and  cost  effectiveness  are  provided  as  feedback  to  the  preliminary 
design  activity  from  the  logical  design  activity.  These  may  arise  from 
detailed  analysis  of  computer  program/equipment  interface  limitations,  refine¬ 
ments  in  estimates  of  the  magnitudes  of  proposed  CPCs,  or  limitations  in 
capabilities  for  achieving  certain  types  of  computational  accuracies. 

Similarly,  inputs  affecting  logical  design  may  be  received  from  the  CPC 
coding  activity.  These  are  normally  in  the  nature  of  computer  programming 
efficiencies,  but  may  also  include  requests  for  design  clarification  or 
requests  for  corrections  due  to  logical  errors. 

A  large  portion  of  the  logical  design  for  utility  computer  programs,  and  to  a 
lesser  extent  for  operational  and  support  computer  programs,  cannot  reasonably 
be  completed  until  the  major  part  of  the  CPC  coding  activity  has  been 
accomplished.  In  the  case  of  utility  CPCs  which  are  used  to  support  the 
development  of  all  other  CPCs  that  comprise  a  CPCEI,  a  need  exists  to  start  the 
CPC  coding  activity  as  early  as  possible.  Because  of  this,  and  because  the 
requirements  for  operating  time,  core  storage  consumption,  and  number  and 
complexity  of  functions  to  be  performed  are  less  critical  than  those  needed 
for  operational  or  support  CPCs,  a  general  and  flexible  initial  statement  of 
logical  design  will  suffice.  The  details  of  design  for  these  CPCs  are  easily 
worked  out  during  the  coding  phase  and  can  be  added  to  the  logical  design 
description  later.  In  those  cases  where  similarly  relaxed  requirements 
for  design  constraints  exist  for  the  operational  and  support  CPCs,  the  final 
design  decisions  can  be  most  expediently  handled  during  the  coding  phase. 

The  following  activities  must  be  accomplished  before  this  phase  of  development 
can  be  completed: 

(1)  For  each  CPC,  provide  a  summary  description  of  the  functions  which 
it  is  to  perform. 

(2)  For  each  CPC,  provide  a  list  of  input  and  output  information 
descriptions  ,  along  with  a  description  of  the  environment  in  which  the  CPC 
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is  to  operate.  Input/output  descriptions  should  include  card  formats,  tape 
formats,  item  size,  table  size,  types  of  data  involved,  and  scaling  information. 
Environmental  descriptions  should  include  names  of  other  CPCs  with  which  the 
subject  CPC  must  interact,  information  concerning  such  things  as  housekeeping 
tasks  which  the  subject  CPC  need  or  need  not  perform,  and  temporary  and  perma¬ 
nent  storage  areas  with  which  it  must  deal. 

(3)  Provide  a  description  of  each  logically  isolable  task  to  be  performed 
within  the  CPC.  These  descriptions  should  include  any  algorithms  or  formulae 

which  are  to  be  used  ,  along  with  statements  of  computational  accuracy  require¬ 
ments.  Descriptions  should  avoid  dictating  coding  techniques,  but  should  be 
of  sufficient  detail  to  describe  what  the  resultant  code  is  to  accomplish. 

(4)  For  each  CPC,  develop  a  logical  flow  diagram  which  shows  the  relation¬ 
ship  of  the  tasks  described  in  3.  above.  All  critical  decision  points  should 

be  included  in  the  flow. 

(5)  For  the  CPCEI  or  each  group  of  logically  related  CPCs  forming  a 
portion  of  the  CPCEI,  produce  a  flow  diagram  showing  the  inter-relationships 
and  sequence  of  operation  of  individual  CPCs. 

(6)  Develop  a  storage  allocation  chart  for  the  CPCEI  or  each  group  of 
logically  related  CPCs  which  form-a  portion  of  the  CPCEI.  Where  a  data 
definition  dictionary  is  to  be  used,  its  content  must  also  be  defined. 

The  level  of  detail  contained  in  the  CPC  logical  design  should  be  sufficient 
to  provide  the  following  types  of  information: 

(1)  Detailed  description  of  interface  relationships  between  CPCs. 

(2)  Detailed  descriptions  of  interface  relationships  between  CPCs  and 
equipment. 

(3)  Supplementary  comments  concerning  special  logic  or  design  structures 
to  be  employed  or  avoided  due  to  transferability  requirements. 

(4)  Error  checks  and  error  message  formats  to  be  incorporated  in  the 
CPC  code. 

(!))  Bookkeeping  requirements. 

( 6 )  Timing  requi rement s . 

(7)  Computational  accuracy  requirements. 

(8)  Critical  decision  points. 
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A-ll. 


Initiate  Development  of  Exercising  Procedures  and  Guides 


The  contractor  responsible  for  the  development  of  the  system  exercising  capability 
should  initiate  at  this  time  the  development  of  descriptive  material  relative 
to  its  use.  This  descriptive  material  should  be  organized  into  the  appropriate 
manuals  and  guides  required  to  train  personnel  at  operational  sites  for 
planning,  preparing,  conducting,  and  evaluating  system  exercises.  Since  the 
system  exercising  capability  will  typically  be  utilized  at  the  Category  II 
test  site,  preparation  of  the  manuals  should  be  complete  prior  to  the  start 
of  the  Category  II  test  activity.  These  documents  are  subject  to  revision 
before  final  issue  to  reflect  the  results  of  Category  II  tests. 

Two  types  of  system  exercises  are  normally  recognized.  One  type  is  oriented 
toward  operational  readiness  training  of  personnel,  the  other  toward  system/ 
subsystem/component  evaluation.  The  distinction  between  the  two  types, 
though  not  always  clearcut,  is  based  on  differences  in  objectives  and  differences 
in  the  nature  and  disposition  of  data  collected.  Separate  manuals  will  normally 
be  prepared  to  cover  each  type  of  exercise. 

A  third  class  of  documents  is  required  to  serve  as  guides  to  identify  the  per¬ 
sonnel  required  to  conduct  exercises  (of  both  types)  and  to  describe  their 
duties,  tasks,  and  responsibilities. 

The  required  contents  of  the  documents  identified  above  are  described 
below: 

(l)  Exercise  Conduct  Manual  -  This  manual  should  present  instruction 
for  planning,  preparing,  and  conducting  simulated  exercises  for  operational 
readiness  training.  Pertinent  information  of  the  following  types  should  be 
presented: 

(a)  Planning  -  Guidance  should  be  provided  relative  to 

(1)  determining  exercise  objectives 

(2)  selecting  configuration 

(3)  specifying  required  simulated  inputs  and  aids 
(h)  identifying  exercise  duty  positions  to  be  manned 
(5)  establishing  implementation  schedules 

(b)  Preparation  -  Methods  and  procedures  should  be  identified  for: 

(1)  preparing  exercise  materials,  e.g.,  generating  tapes, 

scripts,  etc. 

(2)  verification  of  problem  adequacy  and  accuracy 
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(c)  Conduct  -  Methods  and  procedures  for  exercise  conduct  should 
be  described  including: 


(1) 

(2) 

(3) 

operational  system 

(4) ’ 

(5) 


Coordination  with  outside  agencies. 

Briefing  personnel. 

Assuming,  operating  and  relinquishing  control  of  the 

Collecting,  analyzing, and  reporting  performance  data. 
Providing  feedbacks  concerning  the  results  of  the 


exercise  to  the  operational  personnel  exercised. 


(2)  Evaluation  Manual  -  This  manual  is  similar  to  the  Exercise  Conduct 
Manual  in  that  it  is  concerned  with  defining  the  methods  and  procedures  for 
planning,  preparing *  and  conducting  system  exercises.  However,  the  exercises 
to  be  considered  are  not  for  training  operational  personnel,  but  are  for  the 
purpose  of  evaluating  the  operational  readiness  of  the  system  or  some  part 
of  the  system.  The  measures  of  system/subsystem  operational  readiness  to  be 
considered  include: 


(a)  Organizational  effectiveness  against  the  environmental  threat. 

(b)  Adherence  of  personnel  to  operational  procedures  and  doctrine. 

(c)  Proficiency  of  personnel. 

(d)  Effectiveness  of  functional  parts  of  the  system. 

(e)  Effectiveness  of  alternate  configurations  of  personnel, 
equipment,  computer  programs,  doctrine, or  procedures. 

In  general,  the  considerations  discussed  above  relative  to  planning,  preparing, 
and  conducting  training  exercises  also  apply  to  evaluation  exercises.  In 
addition,  special  attention  should  be  directed  to: 

(a)  The  peculiar  requirements  for  evaluation  exercises,  which  should 
be  described  in  detail  and  explicitly  contrasted  with  exercises  conducted 

for  training  purposes. 

(b)  The  nature  of  the  data  to  be  collected  and  the  manner  in 
which  it  will  be  analysed  and  reported.  The  data  from  evaluation  exercises 
should  be  described  in  detail  and  contrasted  with  that  obtained  from 
incidental  observation.  The  use  of  system  capabilities  for  recording,  reducing, 
interpreting, and  reporting  performance  data  during  ordinary  operations  and 
training  exercises  should  be  described.  The  limitations  or  generalizations 
that  can  be  made  on  the  basis  of  such  data  should  be  made  explicit  and  con¬ 
trasted  with  generalizations  that  can  be  made  from  similar  data  collected 
during  exercises  designed  and  conducted  specifically  for  evaluation  purposes. 
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(3)  Exercising  Personnel  Guides  -  These  guides  provide  detailed  informa¬ 
tion  and  job  descriptions  of  the  personnel  required  to  conduct  system  exercises 
for  training  and  evaluation.  Their  intended  purpose  is  to  assist  in  the 
training  of  personnel  who  will  plan  and  conduct  exercises.  Each  exercising 
personnel  duty  position  associated  with  the  exercising  capability  should  be 
identified  and  described  in  detail.  The  description  should  include  the 
following: 


r 


(a)  The  methods  and  procedures  for  overall  management  of  the 
exercising  capability. 


material. 


(b)  The  methods  and  procedures  for  planning  and  preparing  exercise 


(c)  The  methods  and  procedures  to  be  used  in  providing  simulated 
inputs  during  an  exercise. 

(d)  The  type(s)  of  information  available  during  the  mission. 

(e)  The  equipment /facilities  to  be  used. 

(f)  The  positional  responsibilities  and  duties » including  a 
definition  of  the  knowledge  and  skills  required. 

(g)  Structure  of  the  exercising  personnel  organization. 

(h)  Coordination  duties  with  other  personnel/positions. 


Whenever  possible,  the  guides  should  contain  illustrative  examples  to  help 
clarify  the  duties  and  responsibilities  of  the  position  during  typical 
exercise  missions. 

Applicable  Data  Items 

Q-125  Exercise  Conduct  Manual 

Q-124  Evaluation  Manual  (information  System  Exercising  Personnel) 

Q-123  Synthetic  Inputs  Operator  Guides 
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A-12. 


Conduct  Critical  Design  Review  (CDR) 


The  Critical  Design  Review  is  a  formal  technical  reivew  of  the  design  of  a 
CPCEI  for  the  purpose  of  establishing  integrity  of  design  prior  to  coding  and 
testing.  Normally  it  is  accomplished  when  logical  design  is  complete  at  the 
level  of  flow  charts  (Block  A-10).  In  the  case  of  a  complex  computer  program 
CEI  which  is  scheduled  to  reach  any  given  stage  of  the  design/development/test 
process  in  increments  of  computer  program  components  (CPCs)  or  assemblies  of 
CPCs,  the  CDR  may  also  be  scheduled  in  increments.  In  these  cases,  the 
actual  level  of  design  at  which  reviews  are  performed  may  be  adjusted  to 
optimize  the  efficiency  of  the  overall  CDR  for  the  entire  end  item.  For 
example,  when  so  determined  by  the  SPO,  CDRs  may  be  scheduled  in  conjunction 
with  preliminary  qualification  tests  of  components  or  assemblies. 

Normally,  CDRs  are  accomplished  at  the  contractor's  facility  where  the  design 
activity  is  in  progress.  Design  documentation  available  to  support 
the  review  should  include  working  drafts  of  design  information  to  be  incorporated 
subsequently  in  the  Part  II  Specification,  excepting  portions  beyond  the  stage 
of  design  at  which  the  review  is  conducted,  together  with  other  documentation 
describing  results  of  analysis,  preliminary  testing,  etc.,  as  mutually  agreed 
by  the  procuring  agency  and  contractor.  Objectives  of  the  CDR  should  include 
the  following: 

(1)  Establish  compatibility  of  the  design  with  the  Part  I  Detail 
Specification. 

(2)  Establish  system  compatibility  of  the  design,  through  review  of 
design-level  interfaces  among  CPCs  and  with  other  computer  program  CEIs. 

(3)  Review  interactions  with  the  data  base,  e.g.  ,  by  analysis  of 
COMPOOL  tables/listings,  set-used  listings,  etc.,  when  available. 

(1+)  Establish  design  integrity  by  review  of  available  test  and 
analytical  data  in  the  form  of  logic  diagrams,  algorithms,  storage  allocation 
charts,  detail  flow  charts,  etc. 

(5)  Review  functional  interfaces  with  equipment  and  personnel  to 
insure  that  changes  have  not  affected  compatibility. 

The  CDR  should  be  conducted  to  encompass  all  technical  requirements  applicable 
to  the  item  being  reviewed — e.g.,  performance,  operability,  safety,  and 
security.  The  contractor  is  normally  responsible  for  providing  such  necessary 
resources  and  materials  as:  an  agenda  for  the  meeting,  conference  room(s), 
design  documentation  to  be  reviewed,  results  of  supporting  studies /analys is , 
and  minutes  of  the  meeting. 
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Applicable  Data  Item 

ESD  289  Minutes  of  Formal  Reviews  and  Inspections 
References 

AFSCM  375-1,  Exhibit  XIV,  sets  forth  the  formal  requirements  for  design 
reviews  and  inspections  as  they  apply  to  contract  and  items  in  general. 

ESD  Exhibit  EST-3,  Chapter  k,  contains  amplified  information  governing  the 
conduct  of  CDRs  for  CEIs  in  electronic  systems,  including  computer  programs. 


113 


A-13.  Initiate  Coding  of  Computer  Program  Components  (CPCs) 


The  purpose  of  this  activity  is  to  accomplish  the  translation  of  the  Computer 
Program  Components  (CPCs)  logical  design  (Block  A-10)  into  a  series  of 
instructions  which  can  be  transformed  into  the  machine  language  of  the  subject 
computer  by  means  of  an  assembler  or  compiler.  Prerequisites  to  accomplishing 
this  coding  activity  include  the  following: 

(1)  The  CPCEI  preliminary  design  (Block  A-U)  and  the  logical  design  of 
Computer  Program  Components  (Block  A-10). 

(2)  A  description  of  the  computer  and  associated  hardware  which  will 
dictate  coding  characteristics  (e.g.,  scaling,  byte  characteristics). 

(3)  A  specification  of  the  programming  language(s)  to  be  used  for  coding 
the  CPCs. 

(U)  A  storage  allocation  scheme  in  the  form  of  a  data  definition 
dictionary  which  includes  item  and  table  definitions  for  the  CPCEI  data 
base.  Information  contained  in  this  scheme  should  include  (a)  the  location 
and  number  of  registers  allotted  to  each  CPC  for  each  random  access  storage 
device  in  which  it  is  to  reside  and  (b)  for  each  CPC,  a  description  of  the 
content,  size, and  location  for  items  and  tables  to  be  used  for  input  data, 
output  data, and  inter-CPC  communication. 

(5)  A  set  of  conventions  to  be  employed  in  the  development  of  code  for 
the  CPCs  should  be  available.  This  insures  consistency  in  the  form  of  the 
coded  CPCs  and  prohibits  the  development  of  unnecessarily  complex  code  which 
may  be  difficult  to  interpret  or  modify  when  the  need  arises.  Consideration 
should  also  be  given  to  providing  guidelines  for  organizing  computer  code  to 
include  logical  breakpoints  that  can  be  utilized  to  facilitate  test  activities. 

(6)  Additionally,  in  the  case  where  initial  coding  is  to  be  done  using 
an  intermediate  computer,  a  study  must  be  conducted  to  provide  guidelines  for 
coding  techniques  which  should  be  employed  or  avoided,  as  the  case  may  be, 

to  minimize  the  problems  involved  in  transferring  the  CPCs  from  the  inter¬ 
mediate  to  the  final  computer. 

The  CPC  logical  design  is  analyzed  and  interpreted  by  computer  programming 
personnel  and  translated  into  the  language  to  be  used  for  developing  the  CPCEI. 
As  a  result  of  problems  encountered  in  this  task,  or  due  to  special  considera¬ 
tions  for  coding  efficiency,  item  and  table  storage  requirements  in  addition 
to  those  described  above  may  be  defined.  If  storage  allocation  is  to  be  con¬ 
trolled  by  a  data  definition  dictionary, which  is  an  input  to  the  compiler  or 
assembler  used  for  the  development  of  the  CPCEI,  its  translation  to  machine 
language  must  be  accomplished.  In  the  case  where  the  CPCEI  has  but  one  CPC, 
and  no  requirements  exist  for  storage  interface  with  other  CPCEIs ,  storage 
definition  may  be  included  in  the  coding  activity. 
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In  order  to  facilitate  the  assembly  of  the  CPCEI  (Block  A-l8)  and  the  task  of 
testing  the  CPC  (Block  A-lU  and  following)  a  priority  scheme  for  determining  the 
order  in  which  individual  CPCs  are  to  be  coded  must  be  developed.  Such  a 
scheme  insures  the  orderly  development  of  the  CPCEI  and  avoids  situations  where 
work  must  be  delayed  until  the  coding  of  some  critical  CPC  is  completed.  While 
contingency  modifications  to  this  scheme  may  be  made  as  the  coding  activity 
progresses,  its  initial  development  should  occur  before  the  CPC  coding  activity 
is  started.  High  coding  priorities  are  usually  assigned  to  CPCs  having  the 
following  ch aract eristics: 

(1)  CPCs  which  are  to  be  utilized  to  support  the  development  of  other  CPCs. 

(2)  CPCs  which  production  estimates  predict  will  require  the  longest 
linear  time  for  coding. 

(3)  CPCs  which  are  designed  to  have  extensive  interface  with  other  CPCs. 

The  code  for  the  CPC  is  transformed  by  an  assembler  or  compiler  into  the 
machine  language  of  the  computer  to  be  used  with  the  CPCEI.  Errors  resulting 
from  faulty  transformation  of  code  or  errors  in  interpreting  the  logic  of  the 
CPC  design  may  be  discovered  during  this  process.  As  a  result,  several  sets 
of  corrections  must  be  coded  and  a  corresponding  number  of  attempts  to  produce 
a  complete  assembly  or  compilation  may  be  required.  The  goal  of  this  activity 
is  to  produce  a  machine  language  transformation  of  each  CPC  which  is  as  free  from 
error  as  is  possible  and  which  is  acceptable  for  use  in  subsequent  testing 
activities.  As  a  result  of  feedback  from  testing,  the  code  for  the  CPC  must  be 
amended  to  correct  errors.  If  a  symbolic  or  octal  corrector  capability  is  avail¬ 
able,  the  coding  of  corrections  can  be  accomplished  using  these  capabilities  and 
the  need  for  intermediate  compilations  or  assemblies  is  avoided.  If  no  corrector 
capability  is  available,  the  correction  changes  must  be  incorporated  with  the 
code  and  a  new  compilation  or  assembly  performed.  Due  to  errors  of  omission  or 
changes  in  allocation  required  by  error  correction  to  individual  CPCs,  modifica¬ 
tions  to  the  storage  definition  dictionary  may  also  be  required.  In  addition  to 
errors  which  cause  CPCs  to  fail  to  meet  CPCEI  Part  I  Specifications,  various 
programming  inefficiencies  may  be  noted  during  the  test  activity.  These  should 
also  be  corrected  during  this  phase  to  improve  the  quality  of  each  CPC. 

Changes  or  clarifications  which  are  incorporated  in  CPCs  during  the  CPC  coding 
phase  should  be  reflected  in  modifications  made  to  the  CPCEI  preliminary 
design  and  the  logical  designs  of  CPCs.  This  will  facilitate  the  completion 
of  the  CPCEI  Part  II  Specifications  as  described  in  Block  A-23. 

When  the  number  of  persons  assigned  to  work  on  the  coding  of  CPCs  for  CPCEI 
development  is  large,  means  must  be  established  to  maintain  communication  among 
subgroups  of  personnel  and  to  assure  consistency  of  related  efforts  (see  Block 
A-2). 
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Prior  to  the  transfer  of  a  CPC  from  an  intermediate  to  the  final  computer,  the 
CPC  should  be  reassembled  or  compiled  to  incorporate  all  error  corrections 
developed  up  to  the  time  of  transfer.  Additionally,  individual  CPCs  should 
be  assembled  or  compiled  to  incorporate  the  aforementioned  modifications  prior 
to  CPCEI  assembly.  While  further  error  correction  activity  may  occur  during 
subsequent  testing,  the  assembly  of  the  CPCEI  is  greatly  facilitated  by 
having  corrector  patches  incorporated  as  an  integral  part  of  the  components 
to  which  they  apply.  Although  modification  of  CPC  code  may  continue  to  occur 
through  the  stage  of  Adaptation,  Installation  and  Checkout  (Block  A-22) , 
the  basic  coding  phase  may  be  said  to  be  complete  when  all  CPCs  have  passed 
through  the  final  clean-up  assembly  or  compilation. 
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A-l4.  Initiate  Computer  Programming  Test  and  Evaluation 


This  activity  starts  as  soon  as  coding  for  the  first  logically  discrete 
computer  program  routine  is  complete  and  continues  throughout  the  entire 
computer  program  coding  phase.  Objectives  are  to  validate  the  integrity  of 
the  code  with  respect  to  decision  making,  computation,  and  non-arithmetic  data 
processing,  and  to  determine  the  ability  of  the  various  CPCs,  when  assembled, 
to  communicate  with  each  other,  to  operate  as  a  unit,  and  to  correctly  process 
system  inputs  and  produce  system  outputs. 

Computer  programming  test  and  evaluation  progressively  encompasses  three 
distinct  levels  of  testing.  Although,  ideally,  the  requirements  of  one  level 
of  testing  should  be  completely  satisfied  before  proceeding  to  the  next 
higher  level,  in  practice  this  never  occurs.  As  a  result,  activities  at 
lower  levels  are  iterated  as  the  results  of  the  tests  dictate. 

The  lowest  level  of  this  activity  is  a  subprogram  testing.  This  type  of  testing 
is  performed  upon  computer  program  components,  or  perhaps  on  parts  of  CPCs 
when  the  parts  are  logically  independent.  Its  objectives  are  to  determine 
that  each  subprogram  (or  part  thereof):  interprets  its  inputs  correctly; 
performs  correctly  all  functions  assigned  to  it  by  the  design  specifications; 
and  adheres  to  all  conventions,  restrictions,  and  limitations  explicitly  or 
implicitly  defined  in  the  design  of  the  computer  program.  Subprogram  testing 
deals  with  numerous  details  and  demands  punctilious  validation  of  each  detail. 
Each  mathematical  function  must  be  tested  for  accuracy,  using  arguments  not  only 
in  the  middle  range  but  also  at  both  ends  of  the  spectrum  of  values.  Each  major 
decision  point  and  as  many  minor  decision  points  as  possible  must  be  examined 
for  logical  correctness. 

There  exist  two  immediate  prerequisites  to  subprogram  testing.  One  of  these 
is  successful  assembly  or  compilation  of  the  source  subprogram  assuring  the 
technical  accuracy  of  the  code.  The  other  is  visual  examination  of  the  source 
subprogram  assuring  elimination  of  the  more  obvious  logical  errors.  The 
actual  testing  is  an  iterative  process,  for  several  reasons.  The  variety  of 
logical  paths  which  a  subprogram  may  take  cannot  be  tested  in  one  pass. 

Threshold  values  that  might  be  introduced  are  usually  mutually  exclusive. 

Each  error  discovered  and  corrected  requires  a  test  re-run. 

Inputs  to  subprogram  testing  are  simulated.  This  means  that  the  values  which 
are  used  as  inputs  for  subprogram  testing  are  determined  by  the  programmer 
and  are  inserted  by  means  of  a  utility  tool.  The  values  that  are  selected 
should  be  representative  of  the  total  range  of  values  and  conditions  which 
the  subprogram  is  designed  to  process.  Utility  tools  must  be  available  to: 
insert  data;  record  test  outputs;  reduce  test  outputs  to  a  form  amenable  to 
analysis;  and  to  produce  hard  copy  printouts.  For  each  test  that  is  run, 
the  programmer  must  have  prepared  in  advance  a  set  of  expected  results.  The 
actual  results  obtained  by  recording  subprogram  outputs  are  then  compared  with 
expected  results.  Differences  that  exist  indicate  subprogram  errors.  The 
cause  of  the  errors  must  be  isolated,  corrections  made,  and  the  test  re-run. 
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If  no  differences  exist,  it  can  be  assumed  that  the  subprogram  is  error  free. 

The  second  level  of  this  activity  is  functional  area  testing.  This  type  of 
testing  is  performed  upon  a  set  of  one  or  more  CPCs  implementing  a  total 
system  task.  The  objective  of  this  activity  is  to  determine  that:  inputs  to 
the  system  task  are  being  correctly  interpreted;  communications  from  and  to 
the  various  CPCs  relative  to  the  task  are  being  correctly  generated  and  inter¬ 
preted;  and  that  functional  area  outputs  are  being  correctly  generated. 

Functional  area  testing  generally  requires  interaction  with  the  computer 
system  control  program.  In  a  sense,  subprogram  testing  of  the  system  control 
program  may  be  considered  as  taking  place  simultaneously  with  functional  area 
testing.  Analysis  of  test  results  must  always  consider  this  duality,  and 
error  results  must  look  to  both  areas  for  solution. 

System  tasks  interface  with  hardware  and/or  other  system  tasks.  Inputs  to 
functional  area  testing  are  simulated, in  the  sense  that  they  are  programmer 
determined  and  inserted  by  means  of  a  utility  tool.  Inputs  must  be  varied  suf¬ 
ficiently  to  exercise  the  system  task  to  produce  every  class  of  output,  includ¬ 
ing  outputs  due  to  equipment,  operator,  and  program  malfunction.  The  preparation, 
conduct,  and  anlaysis  of  functional  area  testing  is  generally  a  cooperative 
effort,  since  many  computer  programs  (and  programmers)  are  usually  involved. 

As  in  subprogram  testing  the  availability  of  utility  tools  to  insert  data  and 
to  record,  reduce,  and  list  outputs  is  essential.  A  set  of  expected  results 
must  have  been  prepared  in  advance  for  comparison  with  actual  results.  Errors 
are  relatively  easy  to  discern,  but  determination  of  the  error  cause  may  be 
elusive.  This  is  due  to  the  possibility  that  the  error  may  exist  in  either 
inter-program  interface,  control  program  action,  or  action  of  one  or  more  of 
the  computer  programs  constituting  the  functional  area.  A  reversion  to  sub¬ 
program  testing  may  be  necessary  in  order  to  isolate  the  error  before  a  re-run 
of  the  functional  area  test  is  possible. 

The  highest  level  of  this  activity  is  computer  program  system  testing.  This 
type  of  testing  is  performed  upon  a  program  system,  comprised  of  many  programs 
acting  in  concert  to  fulfill  system  goals.  The  objective  of  this  level  of 
testing  is  to  determine  that  the  programs  operate  as  a  unit  in  interpreting 
system  inputs  and  producing  system  outputs.  This  level  of  testing  lays 
emphasis  on;  communications  between  functional  areas;  computer  system  outputs; 
and  the  exercising  of  system  capacities  and  limits. 

System  inputs  are  simulated  and  inserted  by  means  of  a  utility  tool.  The 
availability  of  utility  tools  to  perform  the  function  and  to  reduce  and  list 
system  outputs  and  functional  area  communications  is  assumed.  Errors  discovered 
in  this  level  of  testing  may  cause  reversion  to  functional  area  testing,  or 
even  subprogram  testing.  This  level  of  testing  requires  a  high  degree  of 
planning,  coordination,  and  direction  because  of  the  many  functions  to  be 
tested  and  the  numbers  of  personnel  involved  both  in  the  design  of  the  inputs 
and  analysis  of  outputs. 
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This  total  activity  begins  after  the  Critical  Design  Review  and,  although 
always  a  step  behind,  proceeds  concurrently  with  the  coding  activity.  It 
requires  the  availability  of  a  computer  and  associated  input/output  equip¬ 
ment  (tape  units,  card  reader,  printer,  and  possibly  operator  consoles), 
as  well  as  utility  computer  programs  to  perform  the  functions  of  assembly 
or  compilation,  read-in,  data  insertion,  recording,  data  reduction,  and 
listing. 
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A-15 .  Prepare  Preliminary  Qualification  Test  (PQT)  Procedures 


This  activity  results  in  specifying  the  detailed  procedures  to  be  followed 
in  conducting  a  particular  test  or  set  of  tests  that  are  scheduled  for  the 
purpose  of  implementing  selected  portions  of  the  Category  I  Test  Plan  (Block 
A-9).  Its  objectives  are  to  produce  the  documentation  that  controls  the  conduct 
of  the  tests,  establish  schedules,  and  specify  all  requirements.  It  also 
provides  the  essential  basis  for  PQT  reports  (Block  A-21).  The  Category  I 
Test  Plan  upon  which  the  individual  Category  I  PQT  procedures  are  based 
typically  contains  plans  for  a  series  of  PQTs  and  a  Formal  Qualification  Test 
(FQT).  A  Category  I  Test  Procedures  document  that  is  prepared  in  advance  of 
each  of  these  scheduled  tests  specifies  the  specific  test  objective,  test 
inputs,  expected  outputs,  test  result  analysis  methods,  manning  requirements, 
event  sequence,  and  other  detailed  procedures  for  conducting  the  test. 

Prior  to  conducting  a  PQT  (Block  A-19),  a  Test  Procedures  document  is  prepared 
by  the  contractor  and  submitted  to  the  procuring  agency  sufficiently  in 
advance  of  the  scheduled  test  for  review. 

The  format  and  content  of  a  typical  Test  Procedures  document  are  specified  in 
the  applicable  data  item  to  cover  such  information  as  the  following: 

(1)  The  location  and  schedule  for  the  test,  including  pre-test  briefings 
and  post-test  debriefings  and  data  analysis/reduction. 

(2)  Applicable  reference  documents  such  as  Category  I  Test  Plan,  CEI 
Detail  Specifications,  and  user  documentation. 

(3)  Requirements  and  responsibilities  for  console  operators,  test 
directors,  technical  consultants,  and  other  essential  test  personnel. 

( k )  Requirements  for  computer  programs  other  than  the  CEI  being  tested, 
and  for  equipment  necessary  to  support  the  test. 

(5)  Procedures  for  operating  the  computer  program  to  be  tested,  including 
the  following: 

(a)  Procedures  required  to  read  the  program  into  the  computer, 
establish  the  required  mode,  establish  initial  conditions,  provide  required 
inputs  and  outputs,  and  begin  operation  of  the  computer  program.  Listings  of 
input  material  should  be  provided. 

(b)  Procedures  required  to  maintain  operation  in  those  cases  where 
operator  intervention  is  required. 

(c)  Procedures  for  normal  and  unscheduled  termination  of  program 
operation  and  restarting  procedures. 
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(6)  A  detailed  description  of  test  inputs,  events,  and  expected  results 
related  to  specific  test  objectives,  described  in  the  order  of  occurrence. 

(7)  Requirements  and  procedures  for  reduction  and  analysis  of  test  data, 
including  a  description  of  the  data  to  be  recorded,  means  of  recording,  and 
data  reduction  and  analysis  to  be  accomplished. 

Scheduling  of  test  procedure  development  must  allow  sufficient  time  for: 
selection,  scripting,  generation,  and  verification  of  test  inputs;  development 
of  required  recording  forms  and  associated  documentation  (e.g.,  scripts); 
review,  concurrence,  and  modification  of  the  documents;  scheduling  and 
coordination  of  required  supporting  equipment,  facilities,  communication 
linkages,  and  personnel;  and  updating  of  materials  to  reflect  results  of 
prior  tests,  if  required. 

Applicable  Data  Item 

T-103  Category  I  Test  Plan/Procedures  ( Computer  Programs ) 

Only  the  second  part  (Procedures)  of  the  Form  9  applies 

to  this  item. 
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A-l6.  Update  QQPRI 


This  activity  is  concerned  with  updating  the  preliminary  QQPRI  Report  prepared 
during  Definition  Phase  B  (see  Block  D-33)  to  incorporate  changes  resulting 
from: 


(1)  System  engineering  synthesis  conducted  by  the  SPO  during  Definition 
Phase  C,  which  may  have  altered  equipments,  computer  programs,  and  their 
interfaces  in  such  a  manner  as  to  alter  the  structure  of  duty  positions  and 
associated  manual  and  man-machine  tasks. 

(2)  Firm  definition  of  the  performance/design  requirements  for  operator- 
associated  equipments.  The  QQPRI  Report  prepared  by  the  computer  program 
contractor  during  Definition  Phase  would  have  been  based  in  some  instances  on 
assumptions  regarding  system  hardware  rather  than  on  firm  specifications. 

The  equipment  specifications  should  be  reviewed  and  analyzed  as  required 
to  detail  and  expand  all  duty  position  descriptions  and  related  operator  tasks 
(see  Block  A-5) ,  and  the  results  of  the  analysis  reflected  in  the  updated 
QQPRI  Report. 

(3)  Completion  of  the  CPCEI  Detail  Specifications.  Frequently  it  may 
not  be  possible  to  complete  all  portions  of  the  CPCEI  Detail  Specification 
until  after  the  Acquisition  Phase  contractors  have  been  selected  and  firm 
specifications  for  the  equipment  are  available.  The  completed  CPCEI  Detail 
Specification  must  be  examined  for  possible  impact  on  the  duty  positions 
and  task  descriptions  as  contained  in  the  preliminary  QQPRI  Report. 

(U)  The  development  of  the  Exercise  Problem  Generating  Capability 
(Block  A-6)  and  the  development  of  exercise  procedures  and  guides  (Block  A-ll). 
These  activities  will  provide  detailed  information  relative  to  requirements 
for  personnel  to  prepare,  conduct,  and  evaluate  system  exercises.  Although 
the  basic  duty  positions  for  these  personnel  would  have  been  identified  in  the 
preliminary  QQPRI  Report,  the  additional  information  available  at  this  time 
should  be  incorporated  in  the  updated  report. 

Applicable  Data  Item 

Q-103  Qualitative  and  Quantitative  Personnel  Requirements  Information 
Part  I:  Field  and  Organization  Maintenance 
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A-17. 


Initiate  Preliminary  Qualification  Tests  (PQTs) 


Preliminary  Qualification  Tests  (PQTs)  are  formal  tests  that  are  conducted  by 
the  contractor  in  accordance  with  approved  Category  I  Test  Plans  (Block  A-9) 
and  Category  I  PQT  procedures  (Block  A-15).  PQTs  are  the  means  by  which 
selected  individual  functions  of  a  CPCEI  are  verified  during  the  process  of 
computer  program  design  and  development,  prior  to  the  availability  of  the 
complete  CPCEI  for  formal  qualification  testing.  Typically,  PQTs  consist  of 
progressive  test.s  of  CPCs  or  sets  of  related  CPCEI  functions  as  they  are 
developed,  and  again  as  they  are  integrated  with  other  CPCs  or  functions. 

PQTs  may  be  scheduled  to  be  held  in  conjunction  with  CDR  increments  for  the 
CPCs  or  CPCEI  functions  undergoing  testing.  In  general,  the  activity  should  be 
designed  to  accomplish  the  following  objectives: 

(1)  Provide  the  procuring  agency  with  checkpoints  for  monitoring 
contractor  progress  in  computer  program  development. 

(2)  Permit  detailed  verification  of  CPCEI  functions  that  cannot  be 
throughly  validated  during  FQT  due  to  depth  of  detail  required. 

(3)  Allow  detailed  examination  and  evaluation  of  CPCEI  interfaces  with 
other  CPCEIs. 

(k)  Serve  as  a  demonstration  that  functions  qualified  with  Test  and 
Evaluation  data  are  properly  functioning. 

(5)  When  combined  with  a  CDR,  assure  design  integrity  of  CPC  or  CPCEI 
functions  as  specified  in  Part  I  Detail  Specifications. 

(6)  Permit  detailed  verification  of  adaptation  parameters,  system  limits, 
and  other  critical  values  prior  to  utilization  of  live  supporting  equipments, 
and  personnel,  facilities,  and  communications. 

(7)  Allow  an  orderly  transition  from  the  testing  of  basic  and  elementary 
CPCEI  functions  to  testing  of  the  total  CPCEI ,  to  provide  confidence  that  all 
requirements  and  quality  assurance  provisions  specified  in  Sections  3  and  k  of 
the  Part  I  Detail  Specifications  are  satisfied. 

PQTs  are  preliminary  in  the  sense  that  they  are  not  intended  to  serve  the 
purpose  of  formal  qualification  of  individual  components.  Formal  qualifica¬ 
tion  occurs  later  in  the  Acquisition  Phase  (Block  A-2k) .  CPCEI  acceptance, 
which  involves  acceptance  of  the  Part  II  Detail  Specification  as  an  accurate 
technical  description  of  the  CPCEI,  occurs  after  FQT,  at  FACI  (Block  A-28). 

Since  PQTs  are  preliminary  tests,  they  are  usually  conducted  at  the  contractor's 
design  and  development  facilities  and  witnessed  by  the  procuring  agency.  Prior 
to  test  conduct.  Category  I  PQT  Test  Procedures  are  prepared  and  submitted  for 
review  by  the  procuring  agency  (Block  A-15).  Test  Procedures  documentation, 
in  addition  to  specifying  detailed  operating  instructions,  event  sequences, 
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required  inputs,  and  other  instructional  material,  also  specify  the  test  results 
that  are  expected  or  have  been  predicted.  In  order  to  insure  that  the  computer 
programs  satisfy  the  requirements  of  Part  I  Detail  Specifications,  the  PQT 
expected/predicted  results  are  identified  in  the  Test  Procedures  and  are  directly 
related  on  a  paragraph-by-paragraph  basis  to  the  performance /design  require¬ 
ments  in  Section  3  of  the  Part  I  Detail  Specifications. 

Initial  PQTs  are  generally  oriented  towards  verification  of  detailed  CPC 
operations.  Typical  areas  for  verification  include  subprogram  decision  paths, 
branch  points,  and  data  dictionary  usage.  Also,  detailed  testing  of  timing 
constraints,  storage  space  utilization  as  agreed  to  during  PDRs  ,  and  subprogram 
interfaces  is  best  accomplished  during  the  conduct  of  initial  PQTs.  Test 
input  values  that  are  usually  employed  include  minimums ,  maximums ,  zero 
conditions,  points  of  discontinuity,  system  limits,  selected  illegals,  and 
representative  samples  derived  from  known  critical  value  permutations  and  com¬ 
binations. 

Simulated  test  inputs  are  required  during  PQT  test  conduct.  In  addition,  test 
outputs  must  be  recorded  and  reduced  both  during  and  following  the  tests.  There¬ 
fore,  test  output  and  test  input  tools  to  be  employed  for  these  functions  must 
be  available.  In  most  instances,  if  these  tools  are  deliverable,  they  must  be 
qualified  prior  to  their  employment  in  PQTs.  In  some  instances,  however,  qual¬ 
ification  of  deliverable  test  tools  may  be  inferred  and  demonstrated  by  their 
successful  performance  in  testing  other  CPCEI  functions. 

It  is  desirable  to  conduct  PQTs  using  equipment  which  is  configured  to  meet 
requirements  of  the  given  system,  to  the  extent  feasible.  However,  it  is 
normally  necessary  to  initiate  early  PQTs  with  minimal  (often,  prototype) 
computer  and  peripheral  equipment;  or,  equipment  simulation  may  be  necessary 
during  the  early  part  of  Acquisition  if  a  new  computer,  consoles,  and  associated 
items  are  undergoing  a  parallel  design  and  development  effort.  In  general,  the 
scope  and  realism  of  testing  may  be  progressively  expanded  as  additional  items 
of  the  operational  equipment  are  made  available  for  the  purpose.  Development 
of  the  required  computer  programs  and  the  acquisition  or  provision  of 
satisfactory  substitute  equipment  for  this  task  must  be  evaluated  and  trade-offs 
assessed  within  the  framework  of  cost  constraints,  schedule  implications,  and 
satisfaction  of  PQT  objectives. 

It  is  to  be  expected  that  computer  program  and/or  documentation  errors  will  be 
encountered  during  the  PQT  activity.  Agreement  on  the  requirements  to  be 
satisfied  and  the  procedures  to  be  employed  in  correcting  errors  and  testing 
corrections  must  be  reflected  in  the  appropriate  Test  Procedures  documents. 
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A-18.  Assemble  CPCEI(s) 


The  purpose  of  this  activity  is  to  integrate  individual  CPCs  into  an  assembled 
CPCEI  which  can  then  be  subjected  to  higher  levels  of  testing  and  qualification. 
The  inputs  to  this  phase  are  the  coded  and  tested  CPCs,  including  the  data 
definition  dictionary  (A-lU),  along  with  a  plan  for  assembly  testing  the  CPCEI. 
The  output  is  an  assembled  and  tested  CPCEI  which  should  be  ready  for  the 
Adaptation,  Installation  and  Checkout  (AI&C)  activity  described  below  (Block 
A-22).  In  order  to  complete  this  phase,  the  following  tasks  must  be  accomplished 

(1)  Collect  and  assemble  all  CPCs  which  are  to  comprise  the  CPCEI.  If 

a  machine  language  encoded  data  definition  dictionary  is  a  part  of  the  CPCEI,  it 
is  included  in  this  assembly.  The  assembly  consists  of  loading  all  components 
onto  a  common  peripheral  storage  device  such  as  a  magnetic  tape.  It  is  usually 
performed  in  increments  as  CPCs  which  have  passed  PQT  become  available.  An 
exception  to  this  is  the  data  definition  dictionary  which  is  normally  loaded 
with  the  initial  assembly  but  may  be  recompiled  and  reloaded  as  new  require¬ 
ments  arise  from  the  CPC  code  activity  performed  in  Block  A-lU. 

(2)  Determine  that  all  corrector  patches  produced  in  Block  A-lU  which 
have  not  been  incorporated  with  individual  CPCs  are  added  to  the  assembly 
described  in  (l)  above. 

(3)  Assembly  test  all  functions  related  to  program  areas  associated  with 
interfaces  between  CPCs  to  determine  that  thes  areas  have  been  correctly 
matched. 

(k)  Assembly  test  all  facility  and  input /output  functions  associated  with 
the  CPCEI  to  assure  compatibility  with  available  equipment  and  to  assure  that 
such  functions  are  performed  correctly.  As  these  functions  are  critical  to 
techniques  used  to  assemble  and  modify  the  CPCEI,  their  test  and  correction 
should  receive  first  priority. 

(5)  Add  correctors  or  recompilations  of  CPCs  which  are  developed  to 
correct  errors  discovered  in  the  assembly  test  activity  as  these  become  avail¬ 
able  for  the  base  CPCEI  developed  in  (l)  above. 

(6)  Establish  and  maintain  an  up-to-date  file  of  CPC  and  data  base  inputs 
used  to  produce  the  assembled  CPCEI  along  with  duplicate  back-up  copies  of  the 
CPCEI.  This  provides  for  recovery  from  situations  in  which  the  CPCEI  working 
assembly  used  in  the  assembly  test  activity  is  lost  or  damaged. 

(7)  Record  all  changes  made  to  the  CPCEI  logical  design  which  are  made 
during  this  phase  for  use  in  the  completion  of  CPCEI  Part  II  Specifications. 

CPCEIs  which  are  to  be  used  for  the  support  and  development  of  other  CPCEIs  will 
be  assembled  earliest.  These  include  such  things  as  compilers,  assemblers, 
generators  of  simulated  inputs,  and  data  reduction  tools.  However,  certain 
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functions  of  these  tools  cannot  be  completely  tested  until  the  programs  which 
they  support  have  been  assembled.  A  reduction  in  the  size  of  such  untested 
areas  is  normally  accomplished  by  developing  special  test  programs  which 
simulate  interfaces  internal  to  the  CPCEI.  These  tools  are  developed  by  the 
contractor  as  items  whcih  are  not  necessarily  part  of  the  CPCEI.  Their  use 
is  not  intended  to  obviate  the  need  for  testing  with  real  inputs ,  but  to 
reduce  the  probability  of  unnecessary  delay  in  discovering  critical  errors. 
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A-19. 


Prepare  Preliminary  Qualification  Test  (PQT)  Reports 


Following  the  conduct  of  each  Category  I  Preliminary  Qualification  Test,  the 
contractor  describes  the  test  results  in  a  Category  I  Test  Report.  This 
report  should  be  written  to  satisfy  the  following  procuring  agency  requirements: 

(1)  Identification  of  planned  test  objectives  specified  in  the  related 
Test  Procedure  for  which  test  results  were  identical  with  the  expected 
results,  or  for  which  variations  between  actual  and  expected  results  were  within 
tolerances. 

(2)  Identification  of  planned  test  objectives  for  which  actual  test 
results  differed  from  expected  results. 

(3)  Identification  of  planned  test  objectives  for  which  actual  results 
were  not  obtained,  including  associated  causative  explanations. 

(4)  Recommendations  for  subsequent  action  based  upon  the  identified  test 
results . 

Test  result  materials,  including  such  items  as  computer  and  teletype  printouts, 
completed  switch  action  checklists,  display  verification  sheets  ,  and  other 
materials  required  for  test  result  verification  are  included  in  PQT  Test 
Reports.  Inclusion  of  detailed  test  result  materials  in  the  PQT  Test  Reports 
permits: 

(1)  Valid  and  meaningful  assessments  to  be  made  by  the  procuring  agency  of 
both  the  conduct  of  the  test  and  the  contractor's  recommendations  for  subsequent 
action  based  upon  the  results  of  the  test. 

(2)  Verification  that  the  PQT  was  conducted  in  accordance  with  contractor 
prepared  and  procuring  agency  approved  Test  Plans  and  Test  Procedures. 

(3)  Coordination  with  other  affected  agencies  and  contractors,  if  required, 
to  be  effectively  initiated,  and  detailed  information  regarding  problem  areas 
and  recommended  solutions  to  be  properly  disseminated. 

(k)  Addition  of  meaningful  and  required  information  to  the  contractor's 
documentation  and  data  files. 

Applicable  Data  Item 

T-118  Category  I  Test  Report  ( Computer  Programs ) . 

The  approved  format  for  a  Category  I  Test  Report  is  normally  used 
for  both  PQTs  and  the  FQT.  Standard  content  to  be  covered  includes  the  follow¬ 
ing: 
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(1)  Identification  of  the  CPCEI  to  which  the  Test  Report  applies. 

(2)  Identification  of  the  portions  of  the  associated  Test 
Procedures  to  which  the  Test  Report  applies. 

(3)  The  primary  function(s)  of  the  conducted  test. 

(1+)  Reference  to  both  the  applicable  Test  Plan  and  Test 

Procedures. 

(5)  A  complete  description  of  the  test  results  specifying 
satisfied  objectives,  problem  areas,  and  failures  to  obtain  any  results.  In 
instances  when  actual  test  results  either  differed  from  expected  results  or  no 
results  were  obtained,  reasons  for  the  discrepancies  derived  from  analytical 
processes  must  be  stated.  In  addition,  the  analytical  methods,  techniques,  and 
tools  employed  should  be  specified. 

(6)  Recommendations  for  subsequent  action  based  upon  the  reported 
test  results.  Such  recommendations  may  include,  but  are  not  limited  to: 

(a)  Revising  the  computer  program  code. 

(b)  Revising  the  Part  I  Detail  Specification. 

(c)  Conducting  additional  tests. 

(d)  Qualifying  those  functions  for  which  test  objectives 

were  fulfilled. 

References 

AFSCM  375- *+,  Part  I,  Chap.  U,  Block  l8c 

AFSCM  375-^/ESD  Sup  1,  Part  I,  Chap.  1+,  Blocks  l8d  and  21 
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A-20.  Specify  Initial  Exercising  Problem(s) 


This  activity  is  concerned  with  specifying  the  exercise  problem(s)  to  be  used 
during  the  Category  II  test  program  and/or  initial  exercising  of  operational 
sites.  Although  the  SPO  has  primary  responsibility  for  the  activity,  the  con¬ 
tractor  responsible  for  the  development  of  the  system  exercising  capability  and 
the  production  of  exercise  problem  materials  should  be  an  active  participant. 

The  activity  should  accomplish  the  following: 

(1)  Identify  the  primary  mission  of  the  problem  and  define  specific 
objectives  and  overall  design  concept. 

(2)  Identify  the  simulated  data  that  must  be  generated  to  satisfy  problem 
requirements.  Types  of  input  information  necessary  to  generate  the  simulated 
data  should  be  defined  and  sources  identified. 

(3)  Select  from  the  alternatives  available  the  means  of  inputting  the 
simulated  data.  Where  multiple  input  means  are  selected,  the  simulated  data 
should  be  specifically  allocated  to  each. 

(U)  Identify  and  describe  the  training  aids  and  support  items  that 
must  be  generated. 

(5)  Prepare  a  schedule  of  significant  events  such  as  information  gather¬ 
ing,  dates  of  delivery,  etc.,  and  identify  the  organizations )  responsible 
for  each  event. 

Applicable  Data  Items 

ESD-295  System  Exercising  Problem  Agreements  Document. 

This  Data  Item  is  a  formal  record  of  the  agreements  reached 
between  the  procuring  agency  and  the  contractor  responsible  for  producing 
the  exercising  problem  material.  It  provides  the  basis  for  designing  the 
exercise  problem  and  related  material. 

Q-121  System  Exercising  Problem  Package. 

This  Data  Item  lists  and  describes  the  various  types  of  material 
which  could  be  included  in  any  given  exercise  problem  package.  Although 
specifically  oriented  toward  problem  packages  for  air  defense  systems,  it 
might  also  be  adapted  by  suitable  back-up  sheets  to  define  the  contents  of  a 
problem  exercise  package  for  other  types  of  systems. 
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A-21.  Prepare  Formal  Qualification  Test  (FQT)  Proceudres 


This  activity  is  initiated  by  the  contractor  following  the  preparation  of  the 
Category  I  Test  Plan  (Block  A-9)  that  establishes  the  basis  for  preparation  of 
Category  I  Test  Procedures  for  both  preliminary  and  formal  qualification  of  the 
computer  program  design  requirements  specified  in  paragraph  3  of  the  Part  I 
Detail  Specifications,  The  content  of  the  FQT  Procedures  document  resulting 
from  this  activity  specifies  such  information  as  the  specific  test  objectives, 
test  inputs,  expected  outputs,  test  result  analysis  methods,  manning  require¬ 
ments,  event  sequences,  and  procedures  to  be  followed  in  solving  problem  areas, 
e.g.,  contingency  plans.  Thus,  FQT  Proceudres,  when  completed,  serve  the 
primary  purposes  of  ensuring  effective  control  of  the  FQT,  establishing 
schedules  for  the  FQT  and  all  related  activities,  and  providing  the  basis  for 
the  Category  I  Test  Final  Report  (Block  A-25). 

FQT  Procedures  are  prepared  and  submitted  initially  in  preliminary  form,  since 
approval  by  the  procuring  agency  is  normally  required.  Review  and  analysis  of 
the  preliminary  documentation  may  result  in  proposed  changes  that  must  be 
resolved  and  reflected  in  revisions  prior  to  formal  submission  of  the  FQT 
Procedures.  Therefore,  sufficient  time  must  be  allotted  (e.g.,  three  months 
prior  to  the  scheduled  FQT)  for  this  process  to  be  completed,  including 
necessary  time  for:  selection,  scripting,  generation,  and  verification  of  test 
inputs;  development  of  required  recording  forms  and  scripts;  scheduling  and 
coordination  of  required  supporting  equipment,  facilities,  communication 
ties,  and  personnel;  and  updating  of  materials  as  a  result  of  prior  tests. 

Applicable  Data  Item 

T-103  Category  I  Test  Plan/Procedures  (Computer  Programs). 

Only  Section  2,  Test  Procedures,  of  the  Form  9  applies  to  this 
item.  The  documents  should  cover  the  following  items: 

(1)  The  location  and  schedule  for  the  test,  including  pre-test 
briefings  and  post-test  debriefings. 

(2)  Applicable  reference  documents,  such  as  the  Category  I  Test 
Plan,  Part  I  Detail  Specifications,  and  user  documentation. 

(3)  Requirements  and  responsibilities  of  console  operators,  test 
directors,  technical  consultants,  or  other  essential  test  personnel. 

(4)  Requirements  for  equipment  necessary  to  support  the  test. 

(5)  Procedures  for  operating  the  computer  programs  to  be  tested. 
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(6)  A  detailed  description  of  test  inputs,  events,  and  expected 
results.  This  description  should  he  related  to  specific  test  objectives  in 
the  order  of  occurrence. 

(7)  Requirements  and  procedures  for  reduction  and  analysis  of 
test  data,  including  a  description  of  the  data  to  be  recorded,  means  of 
recording,  and  data  reduction  and  analysis  to  be  accomplished. 
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A-22.  Accomplish  CPCEI  Adaptation,  Installation,  and  Checkout  (AI&C) 


The  activities  of  adaptation,  installation,  and  checkout  of  CPCEI (s)  should 
occur  in  accordance  with  an  approved  plan.  This  activity  is  normally  performed 
following  I&C  of  the  computer  and  associated  equipment  (Block  A-2U).  At  the 
Category  II  Test  site,  this  activity  will  be  accomplished  by  the  computer 
programming  contractor,  following  which  Formal  Qualification  Test  of  the 
CPCEl(s)  will  occur  prior  to  the  initiation  of  Category  II  Test. 

Prerequisite  to  the  task  of  adaptation  to  be  accomplished  at  this  time  is  the 
availability  of  data  base  elements  describing  the  natural  environment  of  the 
Category  II  site  which  had  been  collected  and  compiled  earlier  in  the  Acquisi¬ 
tion  Phase  (Block  A-8).  These  data  must  be  encoded  into  a  form  which  can  be 
recognized  by  the  CPCEl(s)  in  order  to  permit  the  tasks  of  installation  and 
checkout  to  proceed.  For  CPCEIs  which  will  undergo  frequent  or  extensive 
changes  to  this  data  base,  a  supporting  computer  program  which  accomplishes 
the  task  of  encoding  may  be  a  requirement,  in  which  case  it  should  be  avail¬ 
able  at  this  time.  Where  such  a  computer  program  is  not  available,  encoding 
the  data  must  be  done  in  accordance  with  prior  documented  directions  which 
explain  item  and  table  structure,  scaling,  and  the  processes  to  be  followed. 

The  data  must  then  be  assembled  or  compiled  into  the  CPCEI  to  which  they 
apply  and  then  verified.  Verification  of  the  data  is  accomplished  as  part 
of  the  installation  and  checkout. 

Upon  completion  of  the  adaptation  task,  installation  and  checkout  are  accomplished 
employing  the  CPCEl(s),  the  computer  and  peripheral  equipment,  and  other 
associated  equipment  of  the  system.  Checkout  tests  make  use  of  simulated 
inputs  and  live  inputs;  the  degree  to  which  the  latter  can  be  used  will  be 
predicated  upon  the  availability  of  associated  equipments  (e.g.,  radar  ties). 

Although  AI&C  is  a  contractor  responsibility,  assistance  in  the  conduct  of 
tests  may  be  rendered  by  personnel  of  the  using  command  that  are  available  at 
this  time.  Emphasis  is  placed  on  checking  out  the  compatibility  of  the 
CPCEl(s)  with  the  computer  and  other  interfacing  equipments. 

Functions  which  must  be  considered  include  the  following: 

(1)  reading  of  the  CPCEl(s)  from  the  storage  device  (tape,  disc,  cards) 
to  the  computer  operating  element 

(2)  startup 

(3)  switchover  where  applicable 

( ^ )  tape  maintenance  functions 

(5)  representative  switch  actions 
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Checkout  of  the  CPCEl(s)  should  exercise  the  supporting  computer  programs 
as  well  as  the  operational  computer  programs;  many  functions  such  as  simulation, 
recording,  data  reduction,  and  others  of  the  supporting  computer  programs  can 
be  checked  out  by  their  use  in  the  checkout  of  the  operational  computer 
program. 

During  the  course  of  this  activity,  problems  may  be  revealed  which  cannot  be 
readily  ascribed  to  any  particular  source,  that  is,  in  many  cases  the  deter¬ 
mination  of  the  origin  of  a  problem  can  require  extensive  analysis  of  the 
computer  program,  the  computer,  and  other  equipments.  In  such  instances,  a 
means  of  coordinating  efforts  of  experts  in  each  of  the  component  elements 
must  be  established,  with  the  responsibilities  of  each  participant  clearly 
delineated  under  the  direction  of  the  site  activation  coordinator. 

Implicit  in  this  activity  is  the  correction  of  errors  in  the  CPCEI  data  base 
or  computer  program  logic.  Correction  can  be  effected  by  means  of  a  symbolic 
corrector  where  such  capability  exists,  or  by  assembling  or  compiling  the 
CPCEI  to  be  corrected. 

At  subsequent  sites,  adaptation,  installation,  and  checkout  (Block  A-33)  will 
also  follow  the  equipment  I&C,  and  CPCEI  tests  emphasizing  unique-to-site 
adapation  differences  will  be  accomplished  prior  to  Implementation  Tests. 
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A— 23.  Complete  Part  II  CPCEI  Detail  Specifications ) 


The  purpose  of  the  activity  performed  during  this  phase  is  to  produce  a  com¬ 
pleted  set  of  CPCEI  Part  II  Specifications  for  use  in  FACI.  These  specifica¬ 
tions  are  completed  in  accordance  with  the  specifications  for  format  and 
content  defined  in  ESD-237,  Contract  End  Item  Detail  Specification  (Computer 
Program),  Part  II «  In  order  to  complete  this  phase,  the  following  tasks 
must  be  accomplished: 

(l)  A  review  and  update,  if  necessary,  of  CPCEI  design  documentation  to 
insure  that  it  accurately  reflects  the  organization  and  content  of  the  computer 
program  for  the  following  areas : 


(a)  Data  base  storage  allocation  and  item/table  definition. 

(b)  Computer  program  flow  charts. 

(c)  Timing  and  sequencing  characteristics. 

(d)  CPC  interface  with  other  CPCs  and  with  equipment. 


(e)  CPC  limitations  (external  to  those  specified  in  the  CPCEI 
Part  I  Specs ) . 

(f)  CPC  relationship  to  functional  allocations. 

(g)  CPCEI  interface  with  other  CPCEIs. 

(2)  A  review  and  update  of  all  CPCEI  computer  program  listings  to  insure 
that  all  modifications  required  for  error  correction  or  approved  design  changes 
have  been  included. 

(3)  A  review  and  update  of  all  data  base  content  descriptions  for 
completeness  and  accuracy. 


(4)  Collection  of  a  list  of  documents  which  are  applicable  to  the  CPCEI 
Part  II  Specifications  and  which  form  a  part  of  these  specifications. 


(5)  Preparation  of  descriptions  of  the  relationship  of  CPCs  to  the 
CPCEI  data  base. 


(6)  Preparation,  publication,  and  packaging  of  all  CPCEI  Part  II 
Specification  documentation  specified  in  ESD-237* 

(7)  Review  to  insure  conformance  of  the  CPCEI  Part  II  Specification  to 
the  CPCEI  Part  I  Specification. 
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Normally,  the  Part  II  Specification  must  be  completed  and  delivered  sufficiently 
in  advance  of  FACI,  e.g.,  thirty  days,  to  permit  review  and  analysis  by  the 
procuring  agency.  Certain  problems  may  be  occasioned  by  this  requirement  which 
must  be  recognized  and  resolved  through  specific  agreements  relating  to  the 
detailed  specification  content  at  time  of  initial  delivery,  specification 
content  at  FACI,  and  content  of  the  associated  CPCEI  at  each  of  these  times. 

As  examples,  pertinent  considerations  may  include: 

(1)  Whether  actual  listings  of  instructions  and  data  are  included  in 
the  advance  issue  and,  if  so,  whether  or  not  error  corrections  and  other 
interim  changes  (resulting,  for  example,  from  FQT)  are  to  be  incorporated  in 
the  FACI  versions  of  both  the  CPCEI  and  its  specification,  etc. 

(2)  A  cut-off  date  for  incorporating  changes  resulting  from  previously 
approved  ECPs  to  the  Part  I  Specification.  In  some  cases,  the  Part  II 
Specification  changes  may  not  be  scheduled  for  completion  until  after  the 
product  configuration  baseline  is  to  be  established. 

Applicable  Data  Item 

ESD-237  Contract  End  Item  Detail  Specification  (Computer  Program), 

Part  II. 
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A-2U. 


Conduct  Form  ail  Qualification  Test  (FQT) 


Formal  Qualification  Test  (FQT)  is  a  comprehensive  performance  test  of  the 
integrated  CPCEI  to  verify  compliance  with  requirements  of  the  CPCEI  Part  I 
Detail  Specification.  FQTs  are  conducted  by  the  contractor  in  accordance  with 
the  Category  I  Test  Plan  (Block  A-9)  and  Category  I  FQT  Procedures  (Block  A-23). 
The  formal  qualification  of  an  operational  CPCEI  may  normally  be  expected  to 
require  the  use  of  operationally  configured  equipment  which  has  successfully 
passed  FACI.  When  the  required  capabilities  are  not  available  at  an  earlier 
time,  the  computer  program  FQT  may  be  accomplished  at  the  Category  II  test 
facility,  following  installation  and  checkout  of  system  equipment. 

The  requirements  to  be  qualified  at  FQT  are  identified  in  Section  U.1.3  of  the 
Part  I  Specification.  For  a  complex  operational  CPCEI,  the  duration  and 
complexity  of  the  FQT  could  be  prohibitive  unless  the  testing  can  be  designed 
to  emphasize  CPCEI-level  inputs,  outputs,  and  interfaces,  rather  than  detailed 
intermediate  processing  functions.  This  end  can  be  achieved  if  earlier  Pre¬ 
liminary  Qualification  Tests  are  comprehensive  and  successful  in  their  coverage 
of  the  detailed  intermediate  functions,  and  if  assurance  exists  that  design 
integrity  of  the  computer  programs  has  been  maintained  during  testing  of 
successive  CPCEI  assembly  levels.  The  utility  computer  programs  that  are 
used  to  incorporate  error  corrections,  generate  new  assemblies,  and,  in 
general,  modify  computer  programs  after  PQT,  should,  themselves,  be  initially 
qualified  prior  to  such  use. 

While  Preliminary  Qualification  Tests  are  primarily  directed  toward  testing 
of  detailed  CPC  or  functional  area  intermediate  functions  (e.g.,  testing  of 
subprogram  decision  paths,  subprogram  interfaces,  processing  of  specific  data 
items,  and  other  similar  levels  of  testing),  Formal  Qualification  Tests  are 
typically  directed  towards  testing  of: 

(1)  Points  at  which  functional  areas  interchange  data. 

(2)  Processing  of  CPCEI-level  input  messages  received  from  external 
sources  (both  legal  and  illegal). 

(3)  Generation  of  CPCEI-level  output  messages. 

(U)  Integrated  CPCEI  system  limits,  capacities,  and  critical  data  item 
permutations  and  combinations. 

(5)  Critical  timing  factors  and  storage  uses. 

(6)  Computer  program  sequencing  and  conditions  for  transferring 
internal  data  and  computer  programs. 

(7)  Interfaces  with  parallel  operating  computer  programs. 
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A-25.  Prepare  Category  I  Test  Final  Report 


This  report  serves  the  two  purposes  of  (l)  delineating  results  of  the 
Formal  Qualification  Test  and  (2)  summarizing  results  of  the  overall  Category 
I  test  effort  for  the  CPCEI. 

In  satisfying  the  requirements  for  reporting  specific  FQT  results,  the  report 
should  contain  complete  coverage  of  the  following  items : 

(1)  Planned  test  objectives  which  have  been  completely  satisfied, 
through  correspondence  of  actual  test  results  with  predictions  contained 
in  the  FQT  Test  Procedures. 

(2)  Planned  test  objectives  that  were  not  satisfied  due  to  differences 
between  expected  test  results  and  test  results  actually  obtained. 

(3)  Where  discrepancies  exist  between  planned  and  actual  test  methods  or 
results,  the  report  should  contain  analyses  of  causes  and  recommended 
solutions . 

Since  Formal  Qualification  Tests  have  as  their  primary  objective  qualification 
of  the  integrated  computer  program  CEIs  for  Category  II  testing,  an  overall 
summary  status  report  of  the  Category  I  testing  activity  should  be  included 
in  the  Category  I  Test  Final  Report.  Typically,  this  information  will 
consist  of  critical  qualitative  and  quantitative  test  results  and  associated 
objectives  that  have  a  direct  bearing  on  the  Category  II  test  effort,  an 
accounting  of  all  computer  program  modifications  required  prior  to  initiating 
Category  II  testing,  a  summary  of  the  Category  I  testing  effort  with  particular 
attention  given  to  test  objectives  requiring  detailed  verification  during  the 
Category  II  test  effort,  and  an  up-to-date  status  of  the  computer  programs, 
including  all  information  relative  to  the  ability  of  the  subsystem  to  meet  the 
operational  requirements  of  the  System  Performance/Design  Requirements  General 
Specification. 

As  with  PQT  Reports  (Block  A-21),  all  pertinent  FQT  test  result  materials, 
including  such  items  as  computer  and  teletype  printouts,  completed  switch 
action  checklists,  and  display  verification  sheets,  are  included  in  the 
Category  I  Test  Final  Report.  When  properly  prepared,  the  report  will  normally 
be  used  to  serve  the  following  functions: 

(1)  Permit  the  SPO  to  make  valid  and  meaningful  assessments  of  FQT 
conduct  and  results,  in  relation  to  approved  Test  Plans  and  Procedures. 

(2)  Provide  an  adequate  basis  for  assessing  problem  areas  and 
recommended  solutions. 

(3)  Disseminate  significant  test  information  to  participating 
contractors  and  government  agencies. 
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(1+)  Provide  a  record  of  test  methods  and  experiences  for  the  contractor's 
subsequent  use,  as  well  as  for  general  dissemination  to  the  DoD  technical  and 
scientific  community  (see  comment  under  Data  Item  S-17  below). 

Applicable  Data  Items 

T-118  Category  I  Test  Report  (Computer  Programs) 


Requirements  include  the  following: 

(1)  Identification  of  the  CPCEI  to  which  the  Test  Report  applies. 

(2)  Identification  of  associated  Test  Plan/Procedures  and 
specific  test  objectives. 


(3)  Primary  functions  of  the  tests. 

(U)  Complete  description  of  test  results. 

(5)  Summary  and  status  report  of  the  Category  I  test  effort. 

(6)  Recommended  actions  based  on  test  results. 


S- 17-12. 0-1  Technical  Report 

This  data  item  sets  forth  specific  requirements  associated  with 
the  preparation  of  all  reports  to  be  submitted  to  the  Defense  Documentation 
Center  (DDC).  Requirements  typically  apply  to  test  reports. 

References 


AFSCM  375-^>  Part  I,  Chapter  h ,  Block  18c. 

AFSCM  375-^/ESD  Supplement  1,  Chapter  U,  Block  l8d. 
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A— 26 •  Issue  Preliminary  Handbooks  and  Manuals 

The  operator  procedures  developed  during  the  Definition  Phase  (see  Block  D-29) 
and  updated  and  refined  throughout  the  Acquisition  Phase  (see  Blocks  A-5,  A-ll, 
and  A-l6)  are  formally  documented  in  the  form  of  Position  Handbooks,  Users 
Manuals,  and  Exercising  Capability  Manuals  and  Guides.  The  preliminary  copies 
prepared  at  this  time  will  be  used  and  verified  during  Category  II  and  sub¬ 
sequently  reissued  for  operational  use. 

Positional  Handbooks 

A  Positional  Handbook  should  be  prepared  for  each  type  of  operator  position  in 
the  system,  and  should  provide  all  information  necessary  for  performance  of  that 
position.  Although  designed  primarily  as  a  complete  reference  document  to 
supplement  on-the-job  training  and  cross-training,  it  is  also  suitable  for  use 
as  a  basic  text  for  initial  operator  training. 

In  electronic  systems,  the  handbooks  must  typically  meet  the  needs  of  operating 
console  stations  having  computer-generated  displays,  manual  switch  actions, 
and  external  communications  capabilities.  To  be  complete  as  a  basic  reference 
the  handbook  should  include  the  following  types  of  information: 

(1)  A  comprehensive  description  of  the  position  including  positional 
responsibilities  and  duties,  knowledge  and  capabilities  required  of  the  operator, 
and  an  indication  of  where  the  position  exists  in  the  organizational  structure, 
i.e.,  personnel  to  whom  the  operator  is  directly  responsible  and  personnel 
directly  in  his  charge. 

(2)  Detailed  procedures  to  be  followed  in  accomplishing  each  task 
assigned  to  the  position.  Alternate  procedures  should  be  present  where 
applicable  and  any  special  emergencies  procedures  detailed.  Procedural  steps 
should  be  ordered  sequentially  as  they  will  be  performed,  and  information 
presented  explaining  why  the  action  is  taken  and  expected  results.  Factors  that 
should  be  considered  prior  to  taking  any  action  should  be  explained. 

(3)  Content,  format  ,  and  interpretation  of  all  displays  available  to  the 
position  should  be  presented.  Limits,  tolerances,  and  capacities  of  each 
informational  element  displayed  should  be  identified,  including  where 
appropriate  9  the  rate  or  frequency  of  updating. 

(4)  All  switch  actions  available  to  the  positon  should  be  listed  and 
described  in  terms  of  what  the  system  does  in  response  to  the  action  and  how 
it  does  it. 

Operator  Training  Guides 

An  operator  training  guide  should  be  prepared  for  each  major  system  function. 
Examples  of  such  functions  in  air  defense  are:  weapons  commitment  and  control. 
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air  surveillance,  manual  inputs,  real-time  simulation,  etc.  As  a  manual  for 
training  of  system  operators,  including  command  or  supervisory  personnel,  the 
guide  should  present  a  systematic  delineation  of  the  (l)  inputs  to  the  function, 
(2)  data  processing  of  the  information,  including  operator-machine  interaction, 
and  (3)  outputs  from  the  function. 

The  guide  should  be  organized  into  chapters  dealing  with  subfunctions  and 
contain  such  information  as  the  following: 

(1)  General  description  of  the  purpose,  logic  or  objective  of  the 
subfunction  and  its  operational  significance.  Describes  the  kinds  of  data 
being  input  for  processing  and  the  source  from  which  it  is  derived. 

(2)  Description  of  the  automatic/operator-assisted  processing  of 

the  inputs  by  the  computer  program.  Describes  the  switch  actions  and  displays 
available  for  operators  and  the  logic  underlying  their  design  and  usage. 
Describes  operator  responsibilities  under  various  modes  of  operation. 

(3)  Relationship  of  the  subfunction  to  major  functions  and/or  total 
system  operation.  Includes  a  description  of  data  outputs,  especially  those 
involving  system  interfaces. 

The  operator  training  guides  are  oriented  to  functions  and  so  may  cut  across 
positional  duty  assignments.  As  a  basic  text  for  operator  training,  guides 
should  emphasize  wherever  possible  the  concepts  and  logic  underlying  design 
and  operation  with  respect  to  each  function. 

Computer  Program  Users  Manual 

Users  Manuals  are  intended  to  provide  service  and  contractor  personnel  with  the 
necessary  instructions  concerning  usage  of  a  computer  program.  They  should 
contain  the  following  types  of  information: 

(1)  A  definition  of  required  inputs  including  purpose,  use,  input  media, 
limitations/restrictions,  format  and  content,  sequencing,  and  relationship  to 
output . 

(2)  A  description  of  the  results  to  be  expected  after  computer  program 
operation  including  the  form  on  which  results  will  appear,  the  output  format 
and  content,  limitations/restrictions,  and  relationship  to  inputs. 

(3)  Step-by-step  procedures  required  to: 

(a)  Initiate  the  Computer  Program,  including  reading  the  program 
into  the  computer,  establishing  mode  of  operation,  setting  initial  conditions 
(parameters),  and  beginning  operation. 


(b)  Maintain  computer  program  operation  in  those  cases  requiring 
operator  intervention. 

(c)  Terminate  and  restart  the  computer  program  for  normal  and 
unscheduled  interrupts. 

System  Exercising  Capability  Manuals  and  Guides 

System  exercising  manuals  and  guides  are  required  to  describe  the  exercising 
capability  and  manner  in  which  it  should  be  used.  Typically  the  following 
documents  will  be  produced: 

(1)  Exercise  Conduct  Manual — contains  detailed  procedures  for  exercise 
planning,  preparation,  conduct,  and  uses  for  operational  readiness  training. 

(2)  Evaluation  Manual — describes  techniques,  specific  objectives  and 
criteria,  data  reduction  and  evaluation  methods,  and  planning  procedures  for 
system/subsystem  evaluation  exercises. 

(3)  Synthetic  Inputs  Operator  Guide — specifies  duties,  responsibilities, 
and  procedures  for  simulation  personnel  during  the  conduct  of  exercise 
missions . 


Applicable  Data  Items 

ESD-178  Positional  Handbooks — Information  System  Operational  Personnel 
ESD-290  Users  Manual  (Computer  Program) 

Q-125  Exercise  Conduct  Manual 

Q-124  Evaluation  Manual  (Information  System  Exercising  Personnel) 

Q-123  Synthetic  Inputs  Operator  Guide 


A-27.  Complete  Initial  Exercising  Problem  Package(s) 


The  initial  exercising  problems  specified  in  the  activity  described  in  Block 
A-20  must  be  completed  by  this  time  for  use  at  FACI  of  the  computer  program 
(Block  A-28)  and  the  Category  II  test  program. 

The  contents  of  problem  packages  will  vary  widely  depending  upon  the  type  of 
system  involved,  and  within  a  particular  type  of  system  depending  upon  the 
purpose  of  a  specific  exercise.  For  systems  whose  primary  function  is  to 
present  summarized  data  to  command  personnel,  for  example,  the  exercise 
package  might  consist  of  a  series  of  scripts  containing  summary  information 
whose  sequence  of  presentation  would  be  dynamically  related  to  the  decisions  and 
actions  taken  by  the  user  during  the  exercise.  For  real  time  command  and 
control  systems  involving  multiple  sensor  inputs,  complex  data  processing, 
and  personnel  intervention  and  control  of  a  variety  of  effector  mechanisms  #  the 
problem  package  will  have  more  complex  and  diverse  content.  The  problem 
package  for  an  air  defense  system  such  as  SAGE  provides  a  good  example.  Typically 
it  will  contain  a  set  of  correlated  material  designed  for  a  particular  operational 
air  defense  situation.  The  material  providing  the  simulated  radar  inputs  to  the 
system  is  of  major  importance;  this  may  be  provided  in  the  form  of  film  and/or 
magnetic  tape.  Other  problem  aids  are  correlated  with  the  radar  inputs  and 
simulate  other  inputs  normally  received  by  the  system  such  as  flight  plans, 
intelligence  reports,  and  tell  messages.  Some  of  these  inputs  are  routed 
through  regular  operational  channels;  however,  where  this  is  impractical,  a 
script  may  be  utilized  in  simulating  parts  of  the  system.  Still  other  materials 
cite  the  objectives  of  the  problem,  provide;  an  overall  description  of  the 
problem,  and  provide  detailed  descriptions  of  each  flight  included  in  the 
problem.  These  materials  are  reference  aids  for  use  prior  to  the  exercise 
for  planning  and  briefings ;  they  are  used  during  the  exercise  for  monitoring, 
coordinating  and  reporting,  and  after  the  exercise  for  debriefing  and  inter¬ 
preting  the  results.  An  actual  problem  package  for  an  air  defense  system 
exercise  might  contain  any  or  all  of  the  following: 

(1)  Films — contain  radar  target  data  generated  for  each  radar  site 
participating  in  an  exercise,  with  the  data  specifically  oriented  to  each 
participating  site.  Special  equipment  is  required  at  sites  to  convert  the 
data  on  film  into  a  form  suitable  for  processing  by  operational  equipment. 

(2)  Magnetic  Tapes — provide  simulated  data  which,  when  processed  by  the 
operational  computer  programs  ,  produce  a  simulated  air  picture. 

(3)  Maps — provide  pictorial  representatives  of: 

(a)  the  threat  contained  in  the  problem  at  succesive  intervals  of 
problem  time  ; 

(b)  flight  information  on  each  flight  in  the  problem  ,  including 
flight  path,  special  events  along  the  flight  path,  when  in  radar  coverage,  etc.; 


(c)  position  and  heading  of  each  flight  in  a  particular  geographic 
area  at  specified  times  in  the  problem; 

(d)  the  air  picture  as  it  exists  during  each  15,  20,  or  30  minute 
interval  of  problem  time, 

(4)  Map  Overlays — provide  pictorial  representation  of  air  defense  data 
which  do  not  vary  from  problem  to  problem,  such  as  division  boundaries,  etc. 

(5)  Tell  Aids — provide  time  ordered  information  relating  to  threats/ 
tracks  which  would  normally  be  transmitted  to  exercising  sites  and  received 
under  actual  operating  conditions  but  which  will  be  simulated  for  the  purposes 
of  the  exercise. 

(6)  Flight  Plan  Aids-- provide  the  basis  for  the  generation  of  simulated 
flight  plan  information. 

(7)  Jamming  Aids — provide  a  time  and/or  flight  ordered  listing  of  jammer 
flights  defining  the  start  and  end  time,  intensity,  and  direction  of  the  jammer 
with  respect  to  each  site  (for  use  with  films  containing  jamming  targets) 

(8)  Flight  History  List — provides  a  time-ordered  description  of  each 
flight,  including  a  time-ordered  sequence  of  events  along  the  flight  path. 

(9)  Height  Input  Script — provides  altitude  and  location  information  for 
flights,  and  location  information  for  chaff  and  noise  in  the  surveillance  area 
of  each  site. 

(10)  Problem  Description — provides  both  generalized  and  specific  informa¬ 
tion  on  the  problem  objectives  and  problem  content. 

(11)  Reference  Aids — provide  a  source  of  data  which  simulation  personnel 
can  refer  to  during  an  exercise  to  develop  simulated  inputs. 

(12)  Recording  Forms — provide  a  set  of  pre-printed  forms  used  to  aid  in 
the  running  of  the  exercise,  recording  of  results,  etc. 

Applicable  Data  Item 

Q-121  System  Exercising  Problem  Package 

This  Data  Item  lists  and  describes  the  various  types  of  material 
which  could  be  included  in  any  given  exercise  problem  package.  Although 
specifically  oriented  toward  problem  packages  for  air  defense  systems,  it  might 
be  used  with  suitable  back-up  sheets  to  define  the  conuents  of  a  problem 
exercise  package  for  other  types  of  systems. 
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A-28.  Conduct  First  Article  Configuration  Inspection  (FACI) 


First  Article  Configuration  Inspection  is  a  formal  technical  review  and 
inspection  conducted  by  the  SPO.  The  primary  product  of  the  FACI  is  formal 
acceptance  by  the  procuring  agency  of  the  Part  II  Detail  Specification  as  an 
audited  and  approved  document.  Successful  completion  of  FACI  establishes  the 
Part  II  Specification  as  the  Product  Configuration  Baseline  for  the  CEI.  This 
action  also  normally  results  in  the  introduction  of  formal  change  procedures  at 
that  level. 

Since  no  production  lead  time  is  required  for  Category  II  test  or  operational 
inventory,  the  completion  of  FACI  for  a  computer  program  may  not  be  required 
until  late  in  the  acquisition  phase.  For  a  complex  operational  computer  program, 
in  particular,  FACI  should  occur  as  late  in  the  Acquisition  Phase  as  feasible, 
provided  that  it  be  completed  prior  to  acceptance  of  the  item  for  Category  II 
testing.  Thus,  it  will  normally  occur  for  such  CPCEIs  following  installation 
and  checkout,  formal  qualification,  and  completion  of  the  Part  II  Specification. 

Procedures  for  the  control  of  computer  program  changes,  specification  maintenance, 
and  accounting  are  currently  described  in  ESD  Exhibit  EST-1,  May  1966,  Exhibit  IX 
Addendum.  Following  FACI,  these  procedures  will  continue  to  apply  to  the 
processing  of  changes  to  both  Part  I  and  II  of  the  CEI  Specification. 

Applicable  Data  Item 

ESD-289  Minutes  of  Formal  Reviews  and  Inspections 
References 

AFSCM  375-1,  Exhibit  XIV 
ESD  Exhibit  EST-3,  Chapter  5 
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A-29*  Support  Conduct  of  Category  II  Test 


Category  II  testing  is  accomplished  under  Air  Force  direction,  using  the 
integrated  system  in  as  near  the  operational  environment  as  feasible.  In  the 
case  of  an  information  system,  it  is  normally  accomplished  following  installa¬ 
tion  at  the  operational  site  or,  for  a  multi-site  system,  at  the  first  site 
installed.  Primary  objectives  are  to  assure  that  the  system  meets  requirements 
specified  in  the  RAD,  System  Specification,  and  Part  I  Detail  Specification  for 
CEIs. 

The  initiation  of  tests  at  this  time  will  have  been  preceded  by  the  completion 
of  a  comprehensive  Category  II  Test  Plan  and  approved  Category  II  Test  Pro¬ 
cedures  documentation.  Together,  these  documents  define  the  objectives, 
environment,  methods,  instrumentation,  locations,  dates,  conditions,  specific 
operations,  criteria,  and  requirements  for  subsequent  reporting.  Specific 
activities  completed  at  the  Category  II  site  for  an  information  system  will 
have  included  the  installation  and  checkout  of  equipment,  the  adaptation, 
installation,  and  checkout  of  computer  programs  (Block  A-22),  and  the  formal 
qualification  testing  of  operational  computer  programs  (Block  A-2U). 

In  general,  system  engineering  support  of  the  test  activities  consists  of 
participation  in  the  analysis  and  evaluation  of  test  results  and  in  resolving 
technical  problems  encountered.  Reliance  is  normally  placed  on  continued 
support  in  these  areas  by  Acquisition  Phase  contractors  throughout -the 
Category  II  period.  In  the  computer  programming  system  segment,  emphasis  is 
required  on  completing  the  qualification  of  computer  programs  with  respect  to 
conditions  not  previously  available,  e.g.,  live  inputs  and  trained  operating 
personnel,  as  well  as  on  continued  support  to  the  Test  Force  in  diagnosing 
malfunctions,  devising  solutions  to  difficulties  which  Jointly  involve 
computer  programs,  equipment ,  personnel,  and  procedures,  and  formulating  figures 
of  merit  for  subsequent  use  in  Implementation  Testing  (Block  A-3l). 
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A-30.  Accomplish  CPCEI  Adaptation ,  Installation,  and  Checkout  (AI&C)  at 
Subsequent  Sites 

Multi-site  systems  are  normally  activated  on  a  site-by-site  basis,  incrementally, 
following  the  initiation  and  conduct  of  Category  II  testing  at  the  first  site 
installation.  While  the  Category  II  testing  may  not  be  fully  complete  in  all 
cases  prior  to  the  beginning  of  installations  at  subsequent  site  locations,  it 
should  have  progressed  sufficiently  to  provide  necessary  procedures  and 
criteria  for  Implementation  Tests  and  to  complete  the  qualification  of  CEIs. 

In  general,  it  is  assumed  that  CPCEIs  will  have  been  fully  qualified  by  this 
time.  The  Part  II  CPCEI  Specifications  will  have  been  established  as  product 
configuration  baselines,  including  adaptation  data  for  all  sites.  Hence,  the 
AI&C  activity  at  this  time  should  be  relatively  routine,  in  comparison  with 
AI&C  at  the  Category  II  site,  to  the  extent  that  the  earlier  objectives  were 
successfully  accomplished. 

However,  the  encoded  adaptation  data  for  each  site  must  be  properly  incorporated 
into  the  computer  program,  and  verified  for  conformance  with  the  Part  II 
Specification  data  listings.  Following  insertion  into  the  computer,  perfor¬ 
mance  tests  are  made  to  accomplish  a  number  of  purposes  in  preparation  for 
initiating  Implementation  Tests.  These  tests,  in  part,  provide  further  verifica¬ 
tion  of  equipment  installation  and  operating  condition,  as  well  as  verification 
that  the  computer  program  operates  satisfactorily  with  the  given  site  adaptation 
data. 

When  difficulties  are  encountered,  their  diagnosis  may  often  involve  the  same 
types  of  complex  analyses  required  during  earlier  Category  II  testing  to 
determine  causes  and  problem  solutions.  Discrepancies  associated  with  adapta¬ 
tion,  for  example,  may  be  caused  by  errors  in  the  coding,  errors  in  the 
specified  data  values,  or  by  wrong  settings  of  equipment  limits  and  various 
other  factors  in  the  installation  or  functioning  of  equipment  and  data  links 
at  the  site.  Frequently,  changes  to  the  computer  program  prove  to  be  the 
most  feasible  solutions  to  problems  encountered. 

The  computer  program  checkout  activity  will  normally  continue  until  the 
system  is  ready  for  Implementation  Tests  (see  Block  A-3l). 
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A- 31.  Support  Conduct  of  Implementation  Tests 


Implementation  Testing  is  accomplished  only  for  multi-site  electronic  systems, 
at  sites  other  than  the  site  selected  for  Category  II  Test,  Its  primary 
purpose  is  to  demonstrate  to  the  using  command  that  the  production  system  as 
installed  and  equipped  at  each  site  is  functionally  complete  and  ready  for 
operational  use. 

At  each  site.  Implementation  Tests  are  preceded  by  equipment  installation  and 
checkout,  a  period  of  Installation  Testing  (by  GEEIA),  and  computer  program 
adaptation,  installation,  and  checkout  (Block  A-30).  The  types  of  tests/ 
demonstrations  accomplished  during  the  Implementation  Test  period  are  comparable 
to  those  of  Category  II,  although  generally  less  complex  to  the  degree  that 
problems  exposed  during  Category  II  have  been  previously  resolved.  Continued 
system  engineering  support  is  also  comparable,  although  normally  at  a  reduced 
level.  Typically,  emphasis  is  placed  on  verifying  (a)  satisfactory  performance 
of  production- run  equipment  elements,  (b)  operation  of  live  communications 
links,  and  (c)  the  accuracy  of  adaptation  data  values  incorporated  in  the 
operational  computer  programs. 


Figure  k  -  Acquisition  Process 


149 


A-12 


A-7 


ACQUISITION  PHASE - 


FIGURE  4.  ACQUISITION  PROCESS 


- ACQUISITION  PHASE - 

FIGURE  4.  ACQUISITION  PROCESS 


CHAPTER  V 


OPERATIONAL  PHASE 


A.  GENERAL 

While  the  Operational  Phase  begins  when  a  using  command  accepts  the  first 
operational  unit,  the  Acquisition  Phase  continues  until  such  time  as  the 
full  operational  responsibility  for  the  system  has  been  transferred  to  the 
user  and  the  SPO  has  completed  the  delivery  of  updating  changes  resulting  from 
Category  II  testing.  Thus,  there  is  often  an  extended  period  of  transition, 
during  which  engineering,  logistic,  and  operational  responsibilities  are 
assumed  on  an  incremental  basis  by  AFLC  and  a  using  command.  Events  described 
under  Blocks  A- 30  and  A- 31  of  the  preceding  chapter,  which  occur  mostly  during 
this  transition  period,  represent  normal  continuations  of  SPO  Acquisition  Phase 
responsibilities  for  the  system  development. 

Engineering  and  logistic  support  responsibilities  for  system  equipment  are 
transitioned  from  AFSC  to  AFLC.  Subsequent  modifications  are  accomplished 
within  the  scope  of  AFLC  engineering,  except  for  major  changes  (Class  IVB  or  V 
modifications  as  defined  by  AFR  57-*+)  which  require  support  by  an  AFSC  SPO. 

When  so  directed  by  Hq  USAF,  major  changes  are  accomplished  under  the  require¬ 
ments  of  AFR  375-1.  In  these  cases,  the  technical  procedures  and  documentation 
requirements  of  AFSCM  375-5  may  be  applied  in  essentially  the  same  way  as  in 
earlier  phases  of  the  system  life-cycle.  To  the  degree  that  information  pro¬ 
cessing  elements  are  involved  in  Class  IVB  or  Class  V  modifications,  they  are 
also  included  among  the  system  engineering  responsibilities  of  the  AFSC  SPO. 

"Engineering  responsibility",  as  currently  defined  in  AFR  57-*+  and  other 
regulations,  does  not  explicitly  encompass  the  development  and/or  operational 
support  of  computer  programs.  In  many  operational  information  processing 
systems,  corrective  actions  and  incremental  improvements  of  operational 
capabilities  are  accomplished  on  a  continuing  basis  by  changes  to  the  computer 
programs.  Within  the  constraints  of  existing  digital  computing  equipment  con¬ 
figuration  in  a  system,  the  changes  which  can  be  made  range  in  magnitude  from 
simple  error  corrections  to  extensive  additions,  deletions,  and  modifications 
in  operating  functions.  Depending  on  magnitude  and  other  factors,  the  changes 
may  be  accomplished  on-site  or  centrally,  by  in-house  ("Blue  Suit")  capability 
or  contractor  support,  or  by  combinations  of  these.  Definitions  of  standard 
classes  of  these  changes  comparable  to  the  AFR  57-*+  classes  of  system/equipment 
modifications  do  not  currently  exist.  Nor  are  the  Operational  Phase  responsibili¬ 
ties  uniformly  defined  among  AFLC,  AFSC,  and  using  commands.  Hence,  the  minimum 
material  contained  in  this  chapter  is  essentially  limited  to  a  synopsis  of  the 
generalized  system/equipment  modification  process  which  is  outlined  in  Blocks 
102  through  106  of  AFSCM  375-5,  Exhibit  1. 
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B. 


DIAGRAM  OF  THE  OPERATIONAL  PROCESS 


The  diagram  for  this  phase  depicts  a  highly  generalized  sequence  of  major 
milestones  associated  with  significant  system  modifications.  The  sequence 
may  occur,  initially,  during  the  period  of  overlap  between  Acquisition  and 
Operational  Phases,  and  may  be  iterated  for  successive  major  modifications 
throughout  the  operational  life  of  the  system. 


c. 


NARRATIVES 


0-1.  Approved  System/CEI  Modification 

User  experience  with  the  system  during  Category  III  testing  and/or  operational 
use  may  normally  he  expected  to  result  in  establishing  requirements  for  system 
modification.  Such  modifications  may  be  in  the  form  of  product  improvement 
changes,  or  may  be  the  result  of  new  or  modified  operational  requirements. 

For  information  systems,  three  classes  of  modifications  are  often  distinguished: 
(l)  combined  equipment  and  computer  program  changes,  (2)  equipment  changes 
only,  and  (3)  computer  program  changes  only.  All  major  modifications,  which 
are  identified  in  AFR  57-^  as  Class  IV  or  Class  V  changes,  will  typically  be 
combined  equipment /computer  program  changes  and  will  require  system  engineering 
support  by  AFSC  which  will  normally  be  identical  to  the  support  of  previous 
changes  to  established  baselines  during  the  Acquisition  Phase.  These  involve 
analyses  to  determine  and  define  the  total  system  impact,  taking  into  account 
all  elements  of  hardware,  facilities,  computer  programs,  personnel,  and  pro¬ 
cedural  data. 

Computer  program  changes  which  do  not  involve  hardware  are  typically  numerous 
and  frequent  during  the  operational  life  of  a  system.  Depending  on  the 
magnitude  of  the  changes,  they  may  require  iteration  of  appropriate  portions 
of  analysis,  design,  and  development  efforts  described  in  preceding  chapters 
of  this  guide,  taking  account  of  impact  on  personnel  and  procedural  data  in 
the  form  of  supporting  handbooks  and  manuals. 
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0-2.  Develop  Follow-On  Modifications 


The  development  of  follow-on  modifications  to  the  system  requires  the  same 
sequence  of  events  as  previously  outlined  for  the  Acquisition  process  to  the 
degree  indicated  by  the  nature  and  the  magnitude  of  the  modification.  Special 
attention  is  required  during  the  system  engineering  activities  to  develop  data 
defining  the  development  test  requirements  for  the  modification.  Such  testing 
may  be  necessary  because  of  (l)  significant  changes  in  system  capability  or 
changes  in  system  application  not  tested  during  Category  I  or  II  tests  or  (2) 
changes  to  correct  system  deficiencies  for  which  test  articles  were  not  available 
during  Category  I  and  II  tests. 


0-3. 


Plan  and  Conduct  Modification  Testing 


The  system  engineering  activity  should  support  planning,  conduct,  and  analysis 
of  testing  at  both  CEI  and  system  levels  as  appropriate  to  the  modification. 
This  support  should  include  (l)  review  of  the  test  plans  and  procedures  for 
compatibility  with  the  development  test  requirements  described  in  Block  0-2 
preceeding,  (2)  identification  and  resolution  of  problems  arising  during  test 
conduct  and  (3)  test  result  evaluation  to  assure  that  performance  and  design 
requirement  compliance  has  been  demonstrated. 
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0-4.  Update  System  and  CEI  Specifications 


Configuration  management  data  affected  by  engineering  changes  required  to 
accomplish  a  Class  IVB  or  V  modification  program  are  managed  by  the  SPO 
configuration  management  division.  AFSCM  375-1  Exhibits  VII  and  IX  apply  to 
changes  in  specifications  for  equipment  CEIs,  and  Exhibits  VII  and  VIII  to  the 
System  Specification. 

In  the  case  of  computer  program  CEIs,  continued  maintenance  of  the  Part  I 
Specifications  is  a  normal  requirement  throughout  the  life  of  the  system.  All 
changes  managed  by  the  SPO,  for  both  Part  I  and  Part  II  CPCEI  Specifications 
and  associated  items  of  procedural  data,  are  accomplished  in  accordance  with 
Exhibits  IX,  XX,  and  XXI  of  ESD  Exhibit  EST-1.  (In  the  1967  revision  of 
AFSCM  375-1  which  is  currently  in  preparation,  the  corresponding  requirements 
will  be  contained  in  Exhibits  IX,  XX,  and  VII.) 
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